Industry testing and telemetry sharing

Tony Lee Microsoft
Jimmy Kuo Microsoft

We will focus on the following related subjects:

  • Industry sharing of threat meta-data and telemetry
  • Industry testing practice, and methodologies based on telemetry data

The anti-malware has long history of collaboration, from monthly to daily/top threat sharing, incidence response working groups, and now meta-data sharing. The evolution of this collaboration is driven by challenges from the threat landscape which demands threat visibility and efficient analysis, that motivates collaboration on data sharing. Its progress was marked by the IEEE ICSG industry working group works, and its common data exchange schema.

However, the effort has not seen the growth and adoption as expected both in number of participants and the level of sharing. The challenges can be broken down into several key areas,

  • Driving incentives and motivators
  • Disparate data aggregation
  • Data units, conversation and arithmetic operations
  • Complexity in data consumption
  • Unequal exchange/distribution of data points

We will closely examine these underlying challenges and propose a set of actions that industry can take to drive forward the data sharing initiative.

Telemetry data sharing also has a significant role in quality and meaningful industry testing.

Analysis and observation of a number of main industry tests reveals a pattern of test sets dominated by samples with low to zero threat prevalence in the field, which bears little user impact. This test practice incentivizes vendors to spend significant resources on producing less quality detections on malware of little ItW significance, at the same time, resulting in higher FP risks.

Some testers attempt to leverage telemetry data from vendors for sample selection and test score calculation that differentiate samples by prevalence, but all run into similar obstacles,

  • Data source incomplete
  • Lack of data sources
  • Aggregation of disparate data in various format and units
  • Risks of vendors gaming the data to gain advantage in test
  • ...

We will lay out a set of principles based on threat telemetry data that support meaningful test methodologies, as well as conduct case studies on test sets, compare and contrast different selection strategies, and evaluate impact with anonymous product results.

History has shown that industry testing is a collaborative effort by both testers and vendors. While testers leverage both samples and data from vendors they test on, test practice they employ incentivizes and motivates vendor practices. We will also propose industry guidelines that support and promote effective telemetry data sharing and its principle application in industry testing.  digg this! digg this

Quick Links

Should software vendors extend support for their products on Windows XP beyond the end-of-life of the operating system?
Yes - it keeps their users secure
No - it encourages users to continue to use a less secure OS
I don't know
Leave a comment
View 23 comments


Malware Prevalence
Adware-misc |##########|
Java-Exploit |########|
Autorun |#####|
BHO/Toolbar-misc |####|
Conficker/Downadup |###|
 View this month's full report

Virus Bulletin currently has 231,284 registered users.