Skip to content

Measuring Multi-Layer Protection

by Trend Micro Employee on ‎11-12-2010 10:43 AM


pub_domain--moriori--carpenters_rule.png Measuring Security


 Today, most advanced anti-malware

 products use multiple layers of

 protection to defend against attacks.  

 How can the comparative effectiveness

 of these products be measured in a

 way that reflects the actual security

 the products provide to customers? 




Measuring multi-layer protection 


Today's anti-malware products typically use multiple layers of defense against web threats.  This is an advance over traditional AV (circa 2000) that utilized a single layer of defense: scanning files in a file directory, examining the file characteristics of each file (e.g.., name, size, MD5 hash) and deciding if the file matched the characteristic of known malware files.  


Multi-layer anti-malware products use multiple technologies to detect and block malicious code.  In Trend Micro products powered by its Smart Protection Network, e-mail sender IP addresses, source URLs, and executed code behavior are used as well as file characteristics to block malware.  Increasingly, anti-malware vendors are adopting similar protection architectures.


To demonstrate the metrics of multi-layer protection, let's concentrate on the simple case of two layers of protection:  URL source and file characteristics for downloaded malware from the web of the generic form: where is a malicious web site and xyz.exe is a malware file. 




In this simplified two-layer architecture, multi-layer protection is provided for malware files downloaded from the internet.  For file transfers from external drives and within a corporate network, only a single layer is provided.  However, it is estimated that over 90% of malware files are delivered over the internet from an identifiable source URL or an e-mail sender's identifiable IP address.




Measuring multi-layer protection


In our simple two-layer case, there are two "detection rates", one at each layer.  However, unless the complete set of attacks are applied to every layer, the inner layers are exposed only to the threats that passed through the outer layers.  Unless a testing lab states otherwise, we must assume that the ability of an inner layer defense to detect a malware file that was already blocked at an outer layer is unknown and not counted in the inner layer's score.


Typically, we have an overall "end-to-end" protection score.  This is the number of threats making their way through all the anti-malware layers.  It is usually expressed as a percentage of the total number of threats attacking the outermost layer.  In the example, if the Exposure Layer blocks 50% of the threats and the Infection Layer blocks 50% of the threats that make it through the Exposure Layer, then the End-to-End protection score would be 25%.  




Let's take a real example from recent tests on small business anti-malware products at independent lab AV-Test. Three layers of defense are tested in the AV-Test measurements:  Exposure Layer (inspection based on source, Infection Layer (inspection based on content), and Dynamic Layer (inspection based on behavior upon execution).




Test results for small business anti-malware products performed by AV-Test (October 2010).  The following table provides detailed descriptions of the products used in these tests.




Different Labs, Different Layers


Not all independent testing labs define the layers of anti-malware protection the same way.  For instance, NSS Labs measures blocking rates at two layers: "on download" and "on execution".  "On download" is blocking based on either the source URL or file characteristics at the time the browser either calls the URL or downloads the malware file.  "On execution" is blocking based on the behavior of the malware code at the time it is executed in the CPU.





Test results for consumer anti-malware products performed by NSS Labs (September 2010).  A full report of the tests is available at: 


If you read the report, you will notice that the "on execution" layer score is not expressed as we have done here. Rather the "additional contribution" to the overall end-to-end is expressed as a percentage of the original total threats applied at the first layer.  This makes it even more unrealistic to compare protection between products at an inner layer.  


As if multi-layer metrics weren't confusing enough.  


A simple rule:  In general, for multi-layer end-to-end protection tests, only the overall protection (end-to-end) and the first layer of defense can be compared among the tested products.  Scores for the inner layers only let you know how well an individual product does at blocking the residual threats that passed undetected through that product's outer layers.



by Trend Micro Employee on ‎11-12-2010 09:46 PM

Oops!  There is an error in the sentence:


"In the example, if the Exposure Layer blocks 50% of the threats and the Infection Layer blocks 50% of the threats that make it through the Exposure Layer, then the End-to-End protection score would be 25%."


It should be 75% not 25%.  75% is the end-to-end detection rate, 25% is the percent of threats that make it through all the layers.



By using this community you agree to the Participation Guidelines and Terms of Use.

Updated OfficeScan 11.0 Product Support

New Worry-Free Business Security 9.0 Support

Deep Security On Demand: Comprehensive protection for Servers Running on AWS

Trend Micro SafeSync for Business: Securely manage, access and share your files online

Join the 'Bring Your Own Device' Research Project

Join the 'Data Protection' Research Project

Read Message from Trend Micro's CEO - Eva Chen.

About the Author
  • Anthony Arrott is product manager for security analytics at Trend Micro. Among other duties, he coordinates Trend Micro’s participation in external benchmark testing programs that measure the protection commercial security software products provide to their customers. Arrott was Director of Threat Research at anti-spyware vendor InterMute, prior to its acquisition by Trend Micro in 2005. In 2007 Dr. Arrott led the project team for Trend Micro HijackThis v2.0 – enhancing the popular malware diagnostic tool originally developed by Merijn Bellekom. Dr. Arrott earned his degrees at McGill University and M.I.T.
What are other Premium Support Customers talking about?  Learn more

Already a TouchPoint member? Just Sign In