Rebuilding testing for the future
Igor Muttik and James Vignoles McAfee
download slides (PDF)
This presentation discusses several aspects related to testing the ability of security products to detect malware. The
complexity of malware and of the security solutions go up really quickly and we present arguments as to why we believe
that comprehensive QA is no longer viable and why a switch to a more statistical approach is in order.
We look into the problem of compiling a representative 'next-generation' sample test set:
- balancing the test speed with the breadth and depth of testing
- ranking threats and removal or downgrading the rank of legacy threats (e.g. DOS and Word6 viruses)
- removal of short-lived and inactive threats (e.g. spammed downloaders where the site was shut down)
- tracking the history and relationship of malware samples (downloader of what? where from? is the URL still alive? gaming password stealer for WoW or for Zhengtu?)
- excluding most HTMLs (are encrypted URLs malicious code or are they just obfuscated data?)
- downgrade or exclude downloaders for the sake of what they download?
- ranking clean data and false alarms (just like malware clean programs are very much not equal)
- better separation of the malware samples and spam (encrypted URLs could be tricky to classify)
- fair representation of local threats (e.g. could there be too many Brazilian password stealers vs oriental trojans related to gaming?)
We present a topological and percolation model of malware distribution and present arguments as to why the user profile
should be part of the test.
We discuss potential solutions to QA problems:
- running different tests for different user profiles
- organizing collections in 'attack sample groups' rather than individual samples
- collecting telemetry data via 'testing/reporting' plug-ins to security products
- using live telemetry to collect malware execution data (frequencies, geo-location information, etc.)
- using telemetry to rank malware attacks
- standardizing the format of telemetry data and sharing it within the industry
- testing complete security products (e.g. AV bundled with anti-spam, rather than pure AV)
VB2010 will take place 29 September - 1 October 2009 at the Westin Bayshore, Vancouver, BC, Canada.