After more than a decade of experience in the field of anti-malware testing, Virus Bulletin began running an anti-spam testing and certification scheme at the beginning of 2009. The VBSpam test has since become a recognized industry standard.

The test uses Virus Bulletin's live email stream, which is sent to all the filters participating in the test in random order, thus exposing each to exactly the same email stream, in real time. The test measures both the false positive rate and the spam catch rate of the products, and VBSpam certifications are awarded to products that exceed a predefined benchmark based on both measurements.

Awards

VBSpam certification is awarded based on the "final score" of a product, which is defined as the spam catch rate (the percentage of spam blocked) minus three times the false positive rate (the percentage of legitimate email blocked). A product wins an award if the value of (SC - 3*[FP]) exceeds 96.

It should be mentioned that both the email corpus used in testing and the way in which a test is set up could significantly influence the test results; hence an SC rate of 98% recorded a VBSpam test does not necessarily indicate the same as an SC rate of 98% measured by a customer or in another test.

However, because all products are tested under the same conditions and are required to filter the same emails, the performance of two (or more) products in the same VBSpam test can be compared against each other. The VBSpam test therefore gives customers a good picture of how well a product performs relative to its competitors.

(Note: for the first four tests, the award structure and criteria were slightly different; each test report clearly states the criteria used in that particular test.)

Product submission

Any vendors interested in submitting anti-spam products for review are advised to contact Allison Sketchley, VB Sales Executive (+44 1235 544034; allison.sketchley@virusbtn.com) and/or Martijn Grooten, Anti-spam Test Director (+44 1235 540235; martijn.grooten@virusbtn.com).

Project Honey Pot

Virus Bulletin's anti-spam tests make use of a live spam feed harvested and provided by Project Honey Pot, which enables us to send thousands of spam emails per day, from all over the world, in real-time to all participating products. More information about Project Honey Pot can be found on their website.

Testing schedule

VBSpam tests will be carried out six times per year, with the results published in Virus Bulletin magazine and on this website.

The product submission deadline for the next anti-spam test is 25 May 2010. The test will run in June and the results will be published in the July issue of Virus Bulletin, with a summary available on this website.

Methodology

Spam changes and so does anti-spam testing; hence the methodology of the VBSpam anti-spam tests will be tweaked and updated to reflect the latest developments and technologies. However, the methodology will not be altered during the course of a test.

The most up-to-date methodology can be found here, while descriptions of the thought processes behind it can be found in the following introductory articles:

• Introducing VB anti-spam testing (January 2009)
• Anti-spam testing: frequently given answers (February 2009)
• Anti-spam test - first trial run (March 2009)

Test results

• Results of the May 2010 test
• Results of the March 2010 test
• Results of the January 2010 test
• Results of the November 2009 test
• Results of the September 2009 test
• Results of the July 2009 test
• Results of the May 2009 test
• Results of the trial run (April 2009)