Software Security

How Should We Make Software Secure?

University of Washington, Microsoft Research, and Carnegie Mellon University Summer Institute

June 15–18, 2003

Challenge Problem: Security Metrics
submitted by Jeannette Wing

What should we count and what do the numbers mean?

First, some background reading:

Three recent empirical studies raised some interesting questions with respect to fixing bugs and releasing fixes. Their answers are based on statistical models and analyses.

How long do bugs live? “Bugs remain in the Linux kernel an average of 1.8 years before being fixed.”

An Empirical Study of Operating Systems Errors, Andy Chou, Junfeng Yang, Benjamin Chelf, Seth Hallem, and Dawson Engler, ACM Symposium on Operating Systems Principles, October 2001, pp. 73-88.

When do exploits occur? “The data that we extracted confirms the hypothesis in which the vast majority of exploits occur long after patches that would thwart them are available—demonstrating that poor administrative procedures are an enabling factor.”

A Trend Analysis of Exploitations, Hilary Browne, William Arbaugh, John McHugh, and William Fithen, IEEE Symposium on Security and Privacy, May 2001, pp. 214 – 229.

When to patch? “We observe that the risk of patches being defective with respect to time has two knees in the curve at 10 days and 30 days after the patch’s release, making 10 days and 30 days ideal times to apply patches.”

Timing the Application of Security Patches for Optimal Uptime, Steve Beattie, Seth Arnold, Crispin Cowan, Adam Shostack, Perry Wagle, and Chris Wright, LISA XVI, November 2002, pp. 101-110.

Challenge to you:

Devise experiments for answering each of the following (kinds of) questions. Decide what data to collect, what things to count, and how to interpret your numbers.

	Which fix should I install when?
	Is this version of a system “more secure” than the previous? E.g., in what way is Netscape 7.0 measurably more secure than Netscape 6.0?
	Which of systems A and B is "more secure" with respect to a given set of services? E.g., how would you measurably compare Linux and Windows with respect to security?

Feel free to define “more secure” more precisely.

For problems or questions regarding this website contact wing@microsoft.com
Last updated: 04/03/03.