Why bother?

How often have you found yourself in the situation where a bug is found in your code due to an edge case you hadn’t considered when writing your unit tests? Or where your nice original test-driven code base has accumulated cruft through quick fixes being applied to it? If you can honestly say this has never happened to you, then you probably don’t need mutation testing. If you have, though, read on…

What is mutation testing

Mutation testing is the technique of mutating your code and rerunning your unit test suite. If your code is fully test-driven, then there shouldn’t be a meaningful change you can make to your code that doesn’t break the tests. Mutations are small changes like replacing a + with a -, a < with a <=, a true with a false and so on.

How does this fit with TDD?

The fundamental tenet of test-driven development (TDD) is that you write a test to express functionality you want in your code, then write the minimum code required to make this test go green (while keeping the existing ones green) and finally refactor to keep your code base clean. If you really are doing this, then there shouldn’t be a single bit of your code that isn’t necessary to make one or more tests pass. The corollary of this is straightforward: if you change anything in your code, at least one test should fail.

How does NinjaTurtles help?

NinjaTurtles automates this process for you. It operates on your source code one method at a time. First, it identifies which tests in your suite might execute this method, to create a subset of your test suite. Then, it applies a series of mutations to your code, and for each “mutant” produced, it executes this subset of your tests. If at least one test fails, then the mutant is killed and mutation testing has passed. If the subset of your tests all pass, then the mutant survives and mutation testing fails.

This is obviously a time consuming process. NinjaTurtles uses a few tricks to optimise its performance:

  • It operates at the intermediate language (IL) level, so assemblies don’t need to be recompiled for each mutant.
  • Mutants are tested in parallel to maximise use of your machines resources and reduce the elapsed time for mutation testing accordingly.
  • The test suite for each method is narrowed down as far as possible.
  • Testing for each mutant is halted at the first failing test (where the test runner permits it) – mutation testing has already passed at this point.