Covering the global threat landscape

Evaluating anti-virus products with field studies

Fanny Lalonde Lévesque École Polytechnique de Montréal
Carlton R. Davis École Polytechnique de Montréal
José M. Fernandez École Polytechnique de Montréal

The evaluation of anti-virus (AV) products is a vital component in helping the industry develop better products that match the evolving malware threats, and in helping users to make informed decisions about product selection. Traditional evaluation methods involve testing in laboratory environments under various threat scenarios, some more realistic than others. In this paper, we present a first study of an alternative method of product evaluation involving real users. We report on the performance of one AV product in a four-month field study involving 50 users, using their own machines in their normal daily business. In addition, we cross-analyse detection data with user behaviour and demographic characteristics in order to determine what factors are conducive to higher risks of infection. We conclude by discussing options that would allow this methodology to migrate to multi-product evaluations, and become a repeatable and viable alternative to traditional lab-based comparative testing.