Effectively Prioritizing Tests in Development Environment

  • Amitabh Srivastava ,
  • Jay Thiagarajan

MSR-TR-2002-15 |

Publication

Software testing helps ensure not only that the software under development has been implemented correctly, but also that further development does not break it. If developers introduce new defects into the software, these should be detected as early and inexpensively as possible in the development cycle. To help optimize which tests are run at what points in the design cycle, we have built Echelon , a test prioritization system. Echelon prioritizes the application’s given set of tests, based on what changes have been made to the program. Program change information has been used previously to eliminate tests, but these techniques have serious drawbacks that prevent them from being applied in real environments. Also, unlike previous techniques that operate on source code and use data flow techniques that do not scale well to large programs, Echelon utilizes a fast, simple, and intuitive algorithm that works well in practice and operates on binaries. Echelon utilizes a binary matching system that can accurately compute the differences at a very fine granularity between two versions of the program in binary form. Echelon orders the given tests to maximally cover the affected program so that defects are likely to be found quickly and inexpensively. Although the primary focus in Echelon is on program changes, other criteria can be added in computing the priorities. Echelon is part of our test effectiveness infrastructure that runs under the Windows environment. It is currently being integrated into the Microsoft software development process. Echelon has been tested on large Microsoft product binaries. The results show that Echelon is effective in ordering tests based on changes between two program versions.