AV-Test just published the results of their most recent antimalware vendor testing, and they didn't grant Microsoft Security Essentials and Microsoft Forefront Endpoint Protection their "AV-Test Certified" status.
We conduct a rigorous review of the results whenever test results warrant it. We take the protection of our customers very seriously, and the investments we make to do these reviews is an example of that commitment.
Our review showed that 0.0033 percent of our Microsoft Security Essentials and Microsoft Forefront Endpoint Protection customers were impacted by malware samples not detected during the test. In addition, 94 percent of the malware samples not detected during the test didn't impact our customers.
The antimalware world is challenging, for both antimalware companies protecting their customers and for independent testing organizations trying to determine the efficacy of antimalware products. We choose to meet that challenge by prioritizing our protection work based on prevalence and customer impact measures, as Dennis Batchelder discussed in his recent blog post on Customer-focused prioritization. It is also difficult for independent antimalware testing organizations to devise tests that are consistent with the real-world conditions that customers live in; AV-Test shared some of the difficulties and shortfalls in many of the independent industry tests in a presentation they gave at the AVAR (Association of Anti-Virus Asia Researchers) Security Conference in 2012. We agree with them, it is difficult to get the tests right.
This post reviews AV-Test's results and their approach. In-depth details are provided below, but here are some key upfront data points to keep in mind:
- AV-Test reports on samples hit/missed by category. We report (and prioritize our work) based on customer impact.
- AV-Test's test results indicate that our products detected 72 percent of all "0-day malware" using a sample size of 100 pieces of malware. We know from telemetry from hundreds of millions of systems around the world that 99.997 percent of our customers hit with any 0-day did not encounter the malware samples tested in this test.
- AV-Test's test results indicate that our products missed 9 percent of "recent malware" using a sample size of 216,000 pieces of malware. We know from telemetry that 94 percent of these missed malware samples were never encountered by any of our customers.
Here's how AV-Test does their scoring:
Test component | Our score | % |
Protection | 1.5/6.0 | 33% |
Repair | 3.5/6.0 | 33% |
Usability | 5.5/6.0 | 33% |
The 1.5 protection score is the score we focused on. Here's a breakdown of what goes into that score:
Protection component | Description | Files tested | % | Our score | % |
0-day malware | Malware seen for the first time, not to be confused with a previously undisclosed vulnerability | 100 | 28% | 0/1.5 | 50% |
Recent malware | Malware that appeared in the wild over the last 2-3 months | 215,999 | 9% | 0/1.5 | 25% |
Prevalent malware | Widespread malware according to AV-Test data | 5,000 | 0% | 1.5/1.5 | 25% |
During the test, our products didn't detect 28 of the 0-day malware samples, and 9 percent of the recent malware samples. AV-Test uses a minimum bar in their scoring: our results for these two areas fell under that bar. The missed samples in both of these sections were where we focused our analysis, as we wanted to ensure we weren't missing anything impactful to our customers.
When we did our review, we found that our customer-focused processes had already added signatures that protected against 4 percent of the missed samples. These files affected 0.003 percent of our customers.
For the remainder of the missed files, we used a retrospective analysis to see if any of our customers encountered these files. We were looking for files that slipped through because of gaps in our telemetry or file collection process. And we found that 2 percent of these files existed across 0.003 percent of our customers.
The other 94 percent of the samples don't represent what our customers encounter. When we explicitly looked for these files, we could not find them on our customers' machines.
In December 2012, we processed 20 million new potentially malicious files, and, using telemetry and customer impact to prioritize those files, added protection that blocked 4 million different malicious files on nearly 3 million computers. Those 4 million files could have been customer-impacting if we had not prioritized them appropriately.
We continually evaluate and look at ways to improve our processes. We know from feedback from customers that industry testing is valuable, and their tests do help us improve. We're committed to reducing our 0.0033 percent margin to zero.
Joe Blackbird
Program Manager
Microsoft Malware Protection Center