Unit Testing Redux

Thinking back over my post from a year ago, I find that the real reason for most bad unit tests is that people are trying too hard, typically for one of the following reasons:

  • Some folks have drunk the "don't repeat yourself" KoolAid: I agree that not repeating code is a virtue in most cases, but unit test code is an exception: cleverness in a test both obscures the intent of the test and makes a subsequent failure massively harder to diagnose.

  • Others want to avoid writing both tests and documentation: they try to write test cases (almost invariably as "doctests") which do the work of real tests, while at the same time trying to make "readable" docs.

Most of the issues involved with the first motive are satisfactorily addressed in the earlier post: refusing to share code between test modules makes most tempations to cleverness go away. Where the temptation remains, the cure is to look at an individual test and ask the following questions:

  • Is the intent of the test clearly explained by the name of the testcase?

  • Does the test follow the "canonical" form for a unit test? I.e., does it:

    • set up the preconditions for the method / function being tested.

    • call the method / function exactly one time, passing in the values established in the first step.

    • make assertions about the return value, and / or any side effects.

    • do absolutely nothing else.

Fixing tests which fail along the "don't repeat yourself" axis is usually straightforward:

  • Replace any "generic" setup code with per-test-case code. The classic case here is code in the 'setUp' method which stores values on the 'self' of the test case class: such code is always capable of refactoring to use helper methods which return the appropriately-configured test objects on a per-test basis.

  • If the method / funciton under test is called more than once, clone (and rename appropriately) the test case method, removing any redundant setup / assertions, until each test case calls it exactly once.

Rewriting tests to conform to this pattern has a number of benefits:

  • Each individual test case specifies exactly one code path through the method / function being tested, which means that achieving "100% coverage" means you really did test it all.

  • The set of test cases for the method / function being tested define the contract very clearly: any ambiguity can be solved by adding one or more additional tests.

  • Any test which fails is going to be easier to diagnose, because the combination of its name, its preconditions, and its expected results are going to be clearly focused.