Industry testing and telemetry sharing
Tony Lee Microsoft
Jimmy Kuo Microsoft
We will focus on the following related subjects:
- Industry sharing of threat meta-data and telemetry
- Industry testing practice, and methodologies based on telemetry data
The anti-malware has long history of collaboration, from monthly to daily/top threat sharing, incidence response working
groups, and now meta-data sharing. The evolution of this collaboration is driven by challenges from the threat landscape
which demands threat visibility and efficient analysis, that motivates collaboration on data sharing. Its progress was
marked by the IEEE ICSG industry working group works, and its common data exchange schema.
However, the effort has not seen the growth and adoption as expected both in number of participants and the level of
sharing. The challenges can be broken down into several key areas,
- Driving incentives and motivators
- Disparate data aggregation
- Data units, conversation and arithmetic operations
- Complexity in data consumption
- Unequal exchange/distribution of data points
We will closely examine these underlying challenges and propose a set of actions that industry can take to drive forward
the data sharing initiative.
Telemetry data sharing also has a significant role in quality and meaningful industry testing.
Analysis and observation of a number of main industry tests reveals a pattern of test sets dominated by samples
with low to zero threat prevalence in the field, which bears little user impact. This test practice incentivizes vendors
to spend significant resources on producing less quality detections on malware of little ItW significance, at the same
time, resulting in higher FP risks.
Some testers attempt to leverage telemetry data from vendors for sample selection and test score calculation that
differentiate samples by prevalence, but all run into similar obstacles,
- Data source incomplete
- Lack of data sources
- Aggregation of disparate data in various format and units
- Risks of vendors gaming the data to gain advantage in test
We will lay out a set of principles based on threat telemetry data that support meaningful test methodologies, as well
as conduct case studies on test sets, compare and contrast different selection strategies, and evaluate impact with
anonymous product results.
History has shown that industry testing is a collaborative effort by both testers and vendors. While testers leverage
both samples and data from vendors they test on, test practice they employ incentivizes and motivates vendor practices.
We will also propose industry guidelines that support and promote effective telemetry data sharing and its principle
application in industry testing.
del.icio.us digg this