Fighting malware and spam

Retrospective testing - how good heuristics really work

Andreas Marx AV-Test.org

Currently, there are no exact test details available as to how well the heuristics of a virus scanner really work. A number of marketing people still claim that their program's heuristics can detect up to 95% of new viruses. However, a number of existing small-scale tests by Joe Wells, the University of Hamburg, and the University of Magdeburg demonstrate that this cannot be the truth. A more realistic value should be something between 15% and 55% for most scanners - and this explains the need for at least weekly updates.

Currently, there is no exact retrospective test available, but we want to fill this gap now. In order to do this, we have collected all available anti-virus updates of about 20 programs over a period of more than nine months for the program, engine and signature updates (currently 57 GB of compressed Image files).

The results of both an ITW test and a few more zoo tests show how rapidly an anti-virus program becomes outdated and the development of heuristics in the past up to today on a `general' and a `per-product'-basis. The paper will also discuss the limitations of such tests as well as a few conclusions.