Study promotes challenge-response for anti-spam
Interactive system rated best blocker in questioned survey.
A study of anti-spam systems, using a bespoke scale to rate protection offered, has found the challenge-response
method most effective, with a massive lead over managed service systems, rated second place, and with ISP-based
filters trailing far behind the rest of the field. However, with the results receiving widespread coverage,
both the methodology of the study and the neutrality of the researchers has been called into question by
anti-spam experts.
The study was carried out by research and consultancy firm Brockmann & Company, who developed a
'spam index' system to rate the effectiveness of spam filters by measuring factors such as the amount of spam
hitting inboxes and the amount of time spent dealing with each unblocked spam and each false positive. Their system
gave challenge-response systems a rating of 160, with hosted services such as MessageLabs and
Postini rated at 316, appliance solutions 349, software-based gateway filters 366, real-time black-listing
367 and ISP-based filtering worst at 442.
The surprising results have picked up considerable media attention, but have been questioned by some commentators,
including blogger and SpamAssassin developer Justin Mason, who criticised the relative importance given
to false positives, which were given about the same weight in the study as unblocked spam, rather than being counted as a
much more significant problem as they are by many spam analysts. He also points out apparent links between the
head of the research firm behind the tests and challenge-response vendor Sendio, and discusses the problems
of extra traffic created by challenge-response systems contacting non-existent or unconnected addresses to
query spam origins.
Mason's detailed analysis of the study is here, while a
lengthy press release from Brockmann & Company is carried
here and the full study can be accessed (after a registration
process) here.
19 July 2007
Tags:
challenge-response, research, spam.
del.icio.us 
digg this
