Predicting the Reliability of Mass-Market Software in the Marketplace Based on Beta Usage: a Study of Windows Vista and WIndows 7

The traditional approach for predicting post-release software reliability is to emulate customer usage environments and scenarios during in-house testing. However, this approach has limitations for mass-market software systems (MMSS), such as the Windows operating system, because it is impractical to emulate the usage environments and scenarios of millions of users. This paper presents and validates Usage Profile-based Reliability Measurement Calibration (UPRMC), a novel approach for producing accurate post-release reliability predictions using data from beta releases. This work leverages usage and reliability data from multiple releases of a large commercial operating system spanning more than 3 years and more than 3 million users. Results show that UPRMC produces accurate and credible predictions; it has been adopted by the Windows organization.