Fighting malware and spam

VB anti-spam testing - results of the first trial

After months of preparation, discussion and hard work, we are pleased to present the results of the VB anti-spam testing trial. A full description of the trial run including the test setup and methodology can be found here.

Unfortunately, for various reasons we were not able to include all of the 17 products submitted to the test - mainly due to setup issues combined with a tight timeframe. We are still working with the developers of the products excluded from the trial and expect to be able to test them over the course of the next few weeks.

On the test run

The test was run during a period of 11 days in March 2009. During this period, the filters saw a total of 20,764 emails, 877 of which were classified as ham by VB's employees (the recipients). Given the importance of false positives, all emails that appeared to have been falsely reported as spam by any of the products were double-checked to confirm the legitimacy of the email.

In the results below, the spam catch rate, or 'SC rate', is the ratio of the number of correctly classified spam emails relative to the total number of spam emails. Following industry standards, the false positive rate, or 'FP rate', is the ratio of the number of false positives relative to the total number of emails.

Awards

The results of the test are presented below, along with the awards that would have been achieved by the products had this been a 'live' test. Products are anonymised (with the exception of the open source SpamAssassin) for the purposes of this trial run only.

The benchmarks for the awards - at Platinum, Gold and Silver level - are based on the average product scores. Gold and Platinum certificates are awarded to products that perform better than the average - Platinum awards to products that performed twice as well as the average. In order to achieve a silver award a product must achieve a lower than 0.25% FP rate and a higher than 85% SC rate. The same benchmarks are likely to apply to the first 'live' test.

Notes on the results

While most products did very well in blocking spam (especially considering that the test environment was new to them), the false positive rates were surprisingly high almost across the board. The emails concerned have been scrutinized and the majority proved to be emails from mailing lists and newsletters - email that is sent in bulk and notoriously difficult for filters to distinguish from spam. Of course, being difficult for the filters to distinguish is not an excuse for filters to block the mails, and as proved by the FP double-checking process, the blocked emails were all legitimate messages that the recipients wanted to receive - and it is worrying that a relatively large percentage were blocked by the filters.

With this in mind, we have asked the developers involved to look at the configuration of their products. We will review the award benchmarks after each test and alter them (for subsequent tests) if it is deemed necessary (benchmarks will always be fixed before the start of a test).

Results

We have been lenient during the trial run and included the result of products that worked well for only part of the testing period. During the 'live' tests, products will be expected to run continuously (with the exception of circumstances beyond their control).

See also

The following articles from Virus Bulletin's Spam Supplement describe the proposed test setup and methodology. Although normally only available to Virus Bulletin subscribers, the articles have been made available free of charge for registered users of the website:

• Introducing VB anti-spam testing
• Anti-spam testing: frequently given answers
• Anti-spam test - first trial run

Product submission

Any vendors interested in submitting anti-spam products for review are advised to contact Allison Sketchley, VB Sales Executive (+44 1235 544034; allison.sketchley@virusbtn.com) and/or Martijn Grooten, Anti-spam Test Director (+44 1235 540235; martijn.grooten@virusbtn.com).