Test environment

Products must be submitted in the same form as that in which they are available to the end-user. Developers will be permitted to make changes (should they be required) in order for their product to function in the testing environment - but in all instances the VB test team must be made fully aware of such changes.

It is the developers' responsibility to inform the VB test team of any set-up or configuration changes that need to be made in order for the product to work in the testing environment.

Each product will receive all emails from the same fixed IP address; filtering that is based on the connecting IP address thus cannot be used. However, products may perform IP or domain-based filtering using the information in the Received headers that will be added to the emails to reflect the fact that the email has passed through the test network's MTA. If requested by developers, the IP address can also be added to the emails in an X-Forwarded-For header.

As every email is relayed from a fixed IP address, greylisting should be turned off. If an email cannot be delivered, up to five redelivery attempts will be made, but this can be disabled on a per-product basis if required.

Each product, as far as it contains or works together with an MTA, is required to relay the filtered emails to a back-end MTA; the classification should be modified in the header. Email that has not reached the back-end MTA one hour after its original delivery will be considered to have been marked as spam.

Filters hosted locally must use at least two DNS servers; a third will be added where possible.

The VB test team should be provided with a technical contact with both a good knowledge of the product and an understanding of the testing environment.

Classifying

All products are required to classify each email into one of two categories: 'ham' or 'spam'. If a filter uses other classifications (e.g. 'phishing', 'virus', 'possible spam'), the test team must be informed as to whether these classifications are to be considered ham or spam.

Products are not required to check emails for malware, but spam emails containing malware should be marked as spam. Should a ham message contain a malicious attachment, it will be removed from the test set (thus a product will not be penalized for blocking malware).

The 'golden standard' for each email will be decided upon by the recipient, with the exception of emails for which all filters agree: in this case, their classification will be assumed to be correct. The emails that make up the Project Honey Pot corpus will all be assumed to be spam. If there is serious doubt over whether an email is ham or spam, the email will be removed from the test set.

Filters will not be trained.

Filters are not permitted to use any ad hoc rules based on Virus Bulletin's email traffic, e.g. automatically blocking all email in certain languages or alphabets, or whitelisting certain senders.

Email corpus

The email corpus consists of all email sent to the virusbtn.com domain as well as a spam corpus provided by Project Honey Pot and randomly assigned to existing virusbtn.com addresses. All emails will be forwarded in real time. Emails that claim to arrive from @virusbtn.com addresses will be taken out of the test.

The products will receive approximately one email every two seconds.

Each email will be left unchanged with the following exceptions:

• Each email will have a Received header added as follows:

  Received: from xxx.xxx.xxx (HELO yyy.yyy.yyy [12.34.56.78]) by gateway.vbspam.virusbtn.com (qpsmtpd/0.81) with ESMTP; Tue, 12 May 2009 18:20:24 +0100

  where the date is the current date and time, 12.34.56.78 is the IP address of the sending MTA, yyy.yyy.yyy is the domain in the HELO/EHLO command, and xxx.xxx.xxx is its reverse DNS record (defaults to Unknown if no reverse DNS record is found).

• If no Message-ID header is present, one will be added for the vbspam.virusbtn.com domain.

The HELO/EHLO domain of the SMTP envelope will be changed to reflect delivery from a local MTA; the MAIL FROM and RCPT TO values will be the same as those of the original message.

Awards

On completion of the test a false positive (FP) rate (#ham missed/#ham) and average spam catch (SC) rate (#spam caught/#spam) will be computed for each product.

Products whose spam catch rate minus three times the false positive rate (SC - [3xFP]) is greater than or equal to 96% will earn a VBSpam award.

For each product, up to four false positives will be counted per sender, where two senders are considered to be the same if the value of MAIL FROM is the same or very similar and the sending IP address is on the same /24 network.

The full results of each test will be published via a detailed report in the Spam Supplement section of Virus Bulletin magazine (available to all Virus Bulletin subscribers), and a basic summary of the results will be available free of charge to registered users of the Virus Bulletin website.

Once a product has been accepted for testing, it may not be withdrawn from the review without 21 days notice from the vendor - full details are provided in the vendor test agreement (please contact us for a copy of the test agreement).