Measuring Security

[#1]

 Measuring Security

 Today, most advanced anti-malware

 products use multiple layers of

 protection to defend against attacks.  

 How can the comparative effectiveness

 of these products be measured in a

 way that reflects the actual security

 the products provide to customers? 

Measuring multi-layer protection 

Today's anti-malware products typically use multiple layers of defense against web threats.  This is an advance over traditional AV (circa 2000) that utilized a single layer of defense: scanning files in a file directory, examining the file characteristics of each file (e.g.., name, size, MD5 hash) and deciding if the file matched the characteristic of known malware files.  

Multi-layer anti-malware products use multiple technologies to detect and block malicious code.  In Trend Micro products powered by its Smart Protection Network, e-mail sender IP addresses, source URLs, and executed code behavior are used as well as file characteristics to block malware.  Increasingly, anti-malware vendors are adopting similar protection architectures.

To demonstrate the metrics of multi-layer protection, let's concentrate on the simple case of two layers of protection:  URL source and file characteristics for downloaded malware from the web of the generic form:  http://abc.com/xyz.exe where abc.com is a malicious web site and xyz.exe is a malware file. 

In this simplified two-layer architecture, multi-layer protection is provided for malware files downloaded from the internet.  For file transfers from external drives and within a corporate network, only a single layer is provided.  However, it is estimated that over 90% of malware files are delivered over the internet from an identifiable source URL or an e-mail sender's identifiable IP address.

Measuring multi-layer protection

In our simple two-layer case, there are two "detection rates", one at each layer.  However, unless the complete set of attacks are applied to every layer, the inner layers are exposed only to the threats that passed through the outer layers.  Unless a testing lab states otherwise, we must assume that the ability of an inner layer defense to detect a malware file that was already blocked at an outer layer is unknown and not counted in the inner layer's score.

Typically, we have an overall "end-to-end" protection score.  This is the number of threats making their way through all the anti-malware layers.  It is usually expressed as a percentage of the total number of threats attacking the outermost layer.  In the example, if the Exposure Layer blocks 50% of the threats and the Infection Layer blocks 50% of the threats that make it through the Exposure Layer, then the End-to-End protection score would be 25%.  

Let's take a real example from recent tests on small business anti-malware products at independent lab AV-Test. Three layers of defense are tested in the AV-Test measurements:  Exposure Layer (inspection based on source, Infection Layer (inspection based on content), and Dynamic Layer (inspection based on behavior upon execution).

Test results for small business anti-malware products performed by AV-Test (October 2010).  The following table provides detailed descriptions of the products used in these tests.

Different Labs, Different Layers

Not all independent testing labs define the layers of anti-malware protection the same way.  For instance, NSS Labs measures blocking rates at two layers: "on download" and "on execution".  "On download" is blocking based on either the source URL or file characteristics at the time the browser either calls the URL or downloads the malware file.  "On execution" is blocking based on the behavior of the malware code at the time it is executed in the CPU.

Test results for consumer anti-malware products performed by NSS Labs (September 2010).  A full report of the tests is available at:  

http://www.nsslabs.com/research/endpoint-security/anti-malware/consumer-anti-malware-products:-group... 

If you read the report, you will notice that the "on execution" layer score is not expressed as we have done here. Rather the "additional contribution" to the overall end-to-end is expressed as a percentage of the original total threats applied at the first layer.  This makes it even more unrealistic to compare protection between products at an inner layer.  

As if multi-layer metrics weren't confusing enough.  

A simple rule:  In general, for multi-layer end-to-end protection tests, only the overall protection (end-to-end) and the first layer of defense can be compared among the tested products.  Scores for the inner layers only let you know how well an individual product does at blocking the residual threats that passed undetected through that product's outer layers.

[#2]

Oops!  There is an error in the sentence:

"In the example, if the Exposure Layer blocks 50% of the threats and the Infection Layer blocks 50% of the threats that make it through the Exposure Layer, then the End-to-End protection score would be 25%."

It should be 75% not 25%.  75% is the end-to-end detection rate, 25% is the percent of threats that make it through all the layers.