Compatibility Testing via Patterns-Based Trace Comparison

In a world of ever evolving software systems, compatibility testing is a necessity. With clients depending on unpublished yet observable behaviors of systems, the effectiveness of compatibility testing depends on uncovering and testing for such behaviors. This task only gets harder with clean room implementations of new versions of systems.

As a solution, we developed a simple approach to test compatibility of programs in terms of their observable behaviors. The approach compares traces of observed behaviors at identical published interfaces of programs to detect incompatibilities stemming from both the presence of previously unobserved behaviors and the absence of previously observed behaviors. The approach hinges on abstracting traces as sets of structural patterns and linear temporal patterns and comparing these sets of patterns.

We validated the effectiveness of the approach by using it to test compatibility between USB 2.0 and USB 3.0 bus drivers during Windows 8 development cycle. Using the approach to test 14 USB 2.0 devices that were functioning without errors with both USB bus drivers, we uncovered 25 compatibility bugs.