Importance of enough samples in product testing

by Trend Micro Employee on ‎09-22-2010 11:54 AM

 

pub_domain--moriori--carpenters_rule.png Measuring Security

 

 If an insufficient number of samples

 are used in comparative tests,

 differences in results may not

 indicate actual differences among

 the tested products. 

 

 

 

 

Importance of enough samples in product testing 

 

Suppose that if 10,000 samples were tested that product A would score 80% while product B would score 70%.  Then if the products were tested with a random selection of only 10 of those 10,000 samples, these are the probabilities of test results:

 

Score

with 10 samples

Product A

80%

(with 10k samples)

Product B

70%

(with 10k samples)

10 / 10

11%

3%

9 / 10

27%

12%

8 / 10

30%

23%

7 / 10

20%

27%

6 / 10

9%

20%

5 / 10

3%

10%

4 / 10

1%

4%

3 / 10

0.1%

0.9%

2 / 10

0.01%

0.14%

1 / 10

0.0004%

0.0138%

0 / 10

0.00001%

0.00059%

total

100%

100%

 

Even though the most likely score for Product A (80%) is 8 out of 10, there is only a 30% chance that Product A’s score will be 8.  Product A has a 37% chance of scoring higher than 8 out of 10 and a 33% chance of scoring lower than 8/10. 

 

When these probabilities are combined with those for Product B, the chances that Product A (80%) scores higher than Product B (70%) are only about 60%.  In a 10 sample test there is 18% chance the products will have the same score and 22% chance Product B will outscore the superior Product A

 

This difficulty is overcome by increasing the number of samples used in the test.  Figure 1 shows the effect on the 95% confidence interval of increasing the sample size from 10 samples to 100 samples to 1000 samples.  The 95% confidence interval indicates the range over which there is 95% chance that the true score falls within that range.

 

358i19881F335171C095

 

 

Announcements



By using this community you agree to the Participation Guidelines and Terms of Use.


Updated OfficeScan 10.6 Product Support

New Worry-Free Business Security 8.0 Support

Deep Security On Demand: Comprehensive protection for Servers Running on AWS

Trend Micro SafeSync for Business: Securely manage, access and share your files online

Join the 'Bring Your Own Device' Research Project

Join the 'Data Protection' Research Project

Read Message from Trend Micro's CEO - Eva Chen.

About the Author
  • Anthony Arrott is product manager for security analytics at Trend Micro. Among other duties, he coordinates Trend Micro’s participation in external benchmark testing programs that measure the protection commercial security software products provide to their customers. Arrott was Director of Threat Research at anti-spyware vendor InterMute, prior to its acquisition by Trend Micro in 2005. In 2007 Dr. Arrott led the project team for Trend Micro HijackThis v2.0 – enhancing the popular malware diagnostic tool originally developed by Merijn Bellekom. Dr. Arrott earned his degrees at McGill University and M.I.T.
TouchPoint
What are other Premium Support Customers talking about?  Learn more

Already a TouchPoint member? Just Sign In