byaarrott11-12-201010:43 AM - edited 11-12-201010:43 AM
Today, most advanced anti-malware
products use multiple layers of
protection to defend against attacks.
How can the comparative effectiveness
of these products be measured in a
way that reflects the actual security
the products provide to customers?
Measuring multi-layer protection
Today's anti-malware products typically use multiple layers of defense against web threats. This is an advance over traditional AV (circa 2000) that utilized a single layer of defense: scanning files in a file directory, examining the file characteristics of each file (e.g.., name, size, MD5 hash) and deciding if the file matched the characteristic of known malware files.
Multi-layer anti-malware products use multiple technologies to detect and block malicious code. In Trend Micro products powered by its Smart Protection Network, e-mail sender IP addresses, source URLs, and executed code behavior are used as well as file characteristics to block malware. Increasingly, anti-malware vendors are adopting similar protection architectures.
To demonstrate the metrics of multi-layer protection, let's concentrate on the simple case of two layers of protection: URL source and file characteristics for downloaded malware from the web of the generic form: http://abc.com/xyz.exe where abc.com is a malicious web site and xyz.exe is a malware file.
In this simplified two-layer architecture, multi-layer protection is provided for malware files downloaded from the internet. For file transfers from external drives and within a corporate network, only a single layer is provided. However, it is estimated that over 90% of malware files are delivered over the internet from an identifiable source URL or an e-mail sender's identifiable IP address.
Measuring multi-layer protection
In our simple two-layer case, there are two "detection rates", one at each layer. However, unless the complete set of attacks are applied to every layer, the inner layers are exposed only to the threats that passed through the outer layers. Unless a testing lab states otherwise, we must assume that the ability of an inner layer defense to detect a malware file that was already blocked at an outer layer is unknown and not counted in the inner layer's score.
Typically, we have an overall "end-to-end" protection score. This is the number of threats making their way through all the anti-malware layers. It is usually expressed as a percentage of the total number of threats attacking the outermost layer. In the example, if the Exposure Layer blocks 50% of the threats and the Infection Layer blocks 50% of the threats that make it through the Exposure Layer, then the End-to-End protection score would be 25%.
Let's take a real example from recent tests on small business anti-malware products at independent lab AV-Test. Three layers of defense are tested in the AV-Test measurements: Exposure Layer (inspection based on source, Infection Layer (inspection based on content), and Dynamic Layer (inspection based on behavior upon execution).
Test results for small business anti-malware products performed by AV-Test (October 2010). The following table provides detailed descriptions of the products used in these tests.
Different Labs, Different Layers
Not all independent testing labs define the layers of anti-malware protection the same way. For instance, NSS Labs measures blocking rates at two layers: "on download" and "on execution". "On download" is blocking based on either the source URL or file characteristics at the time the browser either calls the URL or downloads the malware file. "On execution" is blocking based on the behavior of the malware code at the time it is executed in the CPU.
Test results for consumer anti-malware products performed by NSS Labs (September 2010). A full report of the tests is available at:
If you read the report, you will notice that the "on execution" layer score is not expressed as we have done here. Rather the "additional contribution" to the overall end-to-end is expressed as a percentage of the original total threats applied at the first layer. This makes it even more unrealistic to compare protection between products at an inner layer.
As if multi-layer metrics weren't confusing enough.
A simple rule: In general, for multi-layer end-to-end protection tests, only the overall protection (end-to-end) and the first layer of defense can be compared among the tested products. Scores for the inner layers only let you know how well an individual product does at blocking the residual threats that passed undetected through that product's outer layers.
Anthony Arrott is product manager for security analytics at Trend Micro. Among other duties, he coordinates Trend Micro’s participation in external benchmark testing programs that measure the protection commercial security software products provide to their customers.
Arrott was Director of Threat Research at anti-spyware vendor InterMute, prior to its acquisition by Trend Micro in 2005. In 2007 Dr. Arrott led the project team for Trend Micro HijackThis v2.0 – enhancing the popular malware diagnostic tool originally developed by Merijn Bellekom. Dr. Arrott earned his degrees at McGill University and M.I.T.
What are other Premium Support Customers talking about? Learn more