Efficient Integration Testing using Dependency Analysis

Although testing starts with individual programs, programs are rarely self-contained in real software environments. They depend on external subsystems like language run time and operating system libraries for various functionalities. These subsystems are developed externally to any given program, with their own test processes. Of course, an uncoordinated change in one of the external subsystems may affect the program’s correctness. Test teams therefore add an integration testing step to their process to ensure that programs will continue to operate with different versions of the external subsystems. As full testing may take days or weeks to run, it is useful to understand how to prioritize these tests. We present an integration testing system to understand and quantify the impact of a change, so test teams can focus their testing efforts on the most likely affected parts of the program. Detecting the impact of a change is a hard problem due to the size and complexity of the control and data dependencies involved. Our new approach is based on a binary dependency framework, MaX, that determines control and data dependencies in a system and represents them in a dependency graph. MaX is designed to work on systems consisting of thousands of binaries and millions of procedures. It constructs the graph in multiple steps to allow the analysis of individual binaries to proceed in parallel. MaX provides simple abstractions for defining systems, and provides a simple programming interface to tools for analysis of the graph. The integration testing system also contains two tools that use MaX to advise test teams. MaxCift quantifies the effect of a change to guide how much testing is likely to be needed. MaxScout prioritizes an existing set of tests based on changes made to external subsystems. All of the tools use a binary code based approach that does not require source code for external subsystems, an important requirement for practical use. MaX runs under the Windows environment and is used by Microsoft product teams. Early results show that the system scales to production software and is effective in guiding testing.