VBSpam testing methodology
This document describes the methodology used for Virus Bulletin's anti-spam comparative tests.
If you are interested in submitting your product for future test, please contact firstname.lastname@example.org for more details.
Products must be submitted in a form in which they are available to the end-user. Developers will be permitted to make changes (should they be required) in order for their product to function in the testing environment - but in all instances the VB test team must be made fully aware of such changes.
It is the developers' responsibility to inform the VB test team of any set-up or configuration changes that need to be made in order for the product to work in the testing environment.
Each product will receive all emails from the same fixed IP address; filtering that is based on the connecting IP address thus cannot be used. However, products may perform IP or domain-based filtering using the information in the Received headers that will be added to the emails to reflect the fact that the email has passed through the test network's MTA. If requested by developers, the IP address can also be added to the emails in an X-Forwarded-For header. There is also the possibility of providing a filter with this information during the SMTP transaction, thus enabling pre-DATA filtering.
As every email is relayed from a fixed IP address, greylisting should be turned off. If an email cannot be delivered, up to five redelivery attempts will be made with 20-minute intervals, but this can be disabled on a per-product basis if required. However, unlike in previous tests this does not include 5xx responses during the SMTP connection; these are considered to mark the email as spam and emails will not be sent.
Each product, as far as it contains or works together with an MTA, is required to relay the filtered emails to a back-end MTA; the classification should be mentioned in the header in a clear and unambiguous way. Email that has not reached the back-end MTA one hour after its original delivery will be considered to have been marked as spam.
Filters hosted locally should use at least two DNS servers.
The VB test team should be provided with a technical contact with both a good knowledge of the product and an understanding of the testing environment.
All products are required to classify each email into one of two categories: 'ham' or 'spam'. If a filter uses other classifications (e.g. 'phishing', 'virus', 'possible spam'), the test team must be informed as to whether these classifications are to be considered ham or spam.
Products are not required to check emails for malware, but spam emails containing malware should be marked as spam. Should a ham message contain a malicious attachment, it will be removed from the test set (thus a product will not be penalized for blocking malware).
Filters will not be trained.
Filters are not permitted to use any ad hoc rules based on properties of the mail streams used in our test, e.g. automatically blocking all email in certain languages or alphabets, or whitelisting certain senders. The testers will perform regular tests to make sure this is not happening.
The email corpus consists of the traffic sent to a number of legitimate mailing lists, as well as spam corpuses provided by Project Honey Pot and Abusix, which are randomly assigned to an address on the vbspamtest.com domain. All emails are forwarded in real time. Most of the mailing list emails have their headers rewritten so that they appear to have been sent from the original sender directly to Virus Bulletin; more on which can be found here.
The products will receive approximately one email every two seconds.
Each email will be left unchanged with the exception that of a Received header added as follows:
Received: from xxx.xxx.xxx (HELO yyy.yyy.yyy [188.8.131.52]) by mx.vbspamtest.com (qpsmtpd/0.81) with [E]SMTP id dddddddddddddddddddd.ppp; Tue, 12 May 2009 18:20:24 +0100where the date is the current date and time, 184.108.40.206 is the IP address of the sending MTA, yyy.yyy.yyy is the domain in the HELO/EHLO command, and xxx.xxx.xxx is its reverse DNS record (defaults to Unknown if no reverse DNS record is found). The string dddddddddddddddddddd.ppp is used by the back-end MTA to uniquely identify both the email and the product.
The HELO/EHLO domain of the SMTP envelope will be changed to reflect delivery from a local MTA; the MAIL FROM and RCPT TO values will be the same as those of the original message.
On completion of the test a false positive (FP) rate (#ham missed/#ham) and average spam catch (SC) rate (#spam caught/#spam) will be computed for each product.
Products whose spam catch rate minus five times the false positive rate (SC - [5xFP]) is greater than or equal to 98 will earn a VBSpam award:
(SC - [5xFP]) ≥ 98
Meanwhile, the VBSpam+ logo is awarded to products that combine a spam catch rate of 99.50% or higher with zero false positives.
For each product, up to four false positives will be counted per sender, where two senders are considered to be the same if the value of MAIL FROM is the same or very similar and the sending IP address is on the same /24 network.
A basic summary of the results will be available free of charge to registered users of the Virus Bulletin website.
A copy of the full report can be purchased by non-subscribers as a standalone article ($19.95 per report) here.
Once a product has been accepted for testing, it may not be withdrawn from the review without 21 days notice from the vendor - full details are provided in the vendor test agreement (please contact us for a copy of the test agreement).
Virus Bulletin currently has 230,970 registered users.