Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Finding Errors in .NET with Feedback-Directed Random Testing

Carlos Pacheco, Shuvendu Lahiri, and Thomas Ball

Abstract

We present a case study in which a team of test engineers at Microsoft applied a feedback-directed random testing tool to a critical component of the .NET architecture. Due to its complexity and high reliability requirements, the component had already been tested by 40 test engineers over five years, using manual testing and many automated testing techniques. Nevertheless, the feedback-directed random testing tool found errors in the component that eluded previous testing, and did so two orders of magnitude faster than a typical test engineer (including time spent inspecting the results of the tool). The tool also led the test team to discover errors in other testing and analysis tools, and deficiencies in previous best-practice guidelines for manual testing. Finally, we identify challenges that random testing faces for continued effectiveness, including an observed decrease in the technique's error detection rate over time.

Details

Publication typeInproceedings
Published inInternational Symposium on Software Testing and Analysis (ISSTA '08)
Pages10
NumberMSR-TR-2008-29
InstitutionMicrosoft Research
PublisherAssociation for Computing Machinery, Inc.
> Publications > Finding Errors in .NET with Feedback-Directed Random Testing