Seeding Bugs to Find Bugs – Mutation Testing Revisited

How do you know your test suite is “good enough”? One of the best ways to tell is mutation testing. Mutation testing seeds artificial defects (mutations) into a program and checks whether your test suite finds them. If it does not, this means your test suite is not adequate yet.

Despite its effectiveness, mutation testing has two issues. First, it requires large computing resources to re-run the test suite again and again. Second, and this is worse, a mutation to the program can keep the program’s semantics unchanged – and thus cannot be detected by any test. Such equivalent mutants act as false positives; they have to be assessed and isolated manually, which is an extremely tedious task.

In this talk, I present the JAVALANCHE framework for mutation testing of Java programs, which addresses both the problems of efficiency and equivalent mutants. First, JAVALANCHE is built for efficiency from the ground up, manipulating byte code directly and allowing mutation testing of programs that are several orders of magnitude larger than earlier research subjects. Second, JAVALANCHE addresses the problem of equivalent mutants by assessing the impact of mutations on dynamic invariants: The more invariants impacted by a mutation, the more likely it is to be useful for improving test suites.

We have evaluated JAVALANCHE on seven industrial-size programs, confirming its effectiveness. With less than 3% of equivalent mutants, our approach provides a precise and fully automatic measure of the adequacy of a test suite – making mutation testing, finally, applicable in practice.

Joint work with Valentin Dallmeier, Bernhard Grün, and David Schuler.