The true value of unit testing.

I have been somewhat dismayed recently by prominent developers stating that they don’t do unit testing. Often, the reasons given seek to undermine and diminish the value of the tests, but appear to lack an understanding of the actual value of unit testing.

In this post, I’d like to explore the values of unit testing, and see if the reasons not to test actually stack up. There are, to my way of thinking, three significant values to unit testing. Usually, only two of these values are recognized, particularly by those that don’t test, so I’ll start with those.

Few would dispute the fact that the design of a piece of software can be driven from designing tests for it first. Often the design flows from those tests. Writing tests ahead of writing your code will cause you to think in a more granular way about the interfaces, classes and methods that you’ll need, but that is not the only design value. Critically, in order to write unit tests, you become the consumer of your own code. This might cause you to add utility methods to make classes easier to work with, or might cause you to think about how your interface may be misused.

For example, suppose you have a class which must be used in a particular order. You must first call the class constructor to instance it, and not wanting to raise exceptions in the constructor, you then call an initialize method to initialize that class. What happens if you try to call other methods before initialize? While writing the unit test you may decide to create a factory method somewhere, to make instancing and initializing this class easier.

To give one more example, I recently wrote an object oriented wrapper around the OpenCL API.
Part of the wrapper has a method which returns a list of compute capable devices attached to the system. These devices may be used to allocate buffers, perform computation tasks, etc. My first thought for returning a list of devices was to simply return IList< IDevice >, but this has problems. For one thing, IList<> has an Add() method to add items, but my list needs to reflect the physical hardware attached to the device only, what would happen if someone tried to add an item to the list? This lead to a redesign using an IReadOnlyList<>, but also to a division of the IDevice interface into two, one would return device information, and the other provide access to the device functionality.

The above are just two small examples. I’ve found that being forced into consuming my own code as I write it will lead to a weft of design choices. However, the argument against needing tests to drive design is a reasonably sound one. It goes like this: “I’m now experienced enough that I know to design my code this way, and don’t need the tests to guide me into it.” – This is fair, I’ve experienced it myself, and generally I’ll write code without tests, and retroactively apply tests for this reason, but I still write them! Why? Let’s look at some more of the values of unit testing…

Prevention of Bugs.
Preventing bugs from occurring in your code before, or as you write it, would appear to be the most obvious benefit of unit testing. That said, I’ve heard some compelling arguments against this value too…

“If I am the one writing the test and therefore know what conditions I’m testing for, surely I can simply write code to avoid those conditions in the first place.”

This argument misses a large part of the point of unit testing, because, the fact is that the act of writing the test will illustrate issues you hadn’t already considered! It comes back to the same principal as that of design, it forces you to actually use your code and test it’s extremes, rather than to simply assume you got it all right the first time. I couldn’t count the number of times I’ve had a unit test that I expected to succeed, fail, only to discover I’d missed something trivial in the code under test.

I recently watched a 2015 GOTO conference with Dave Thomas explaining that he’d run statistics on his code and bug capture, and determined that unit testing did not help him to prevent any more bugs than writing code without tests. Well, this might be the case for Dave, but I’ve had a different experience. Sure, I can write code and make it work without tests, where the working resulting product IS the test, and if it does as it should, it works. This does not make the code bug-free, and it demolishes code-reuse. Code-reuse depends on code always doing what it should, in every use case, and not performing as it should in only one use case, and that’s the value of unit testing for bug prevention.

Still, lets suppose you are the most diligent engineer the world has ever seen, and that unit tests really don’t help you to prevent bugs. There is one more value to unit tests which really sets the deal for me…

Preventing breaking changes.
When I write unit tests, I do so with the other benefits in mind, but this benefit as the sole reason. My tests are built and executed by a CI server every single time I commit to the source repository. If I break some other use case while changing my code, the unit tests running on the CI server is my last opportunity to catch the breaking change.

You may have automated testing, and QA testing after your commit, but the way I view breaking change bugs is that if it got to another round of testing, then I, as the engineer that wrote the code, have caused churn. If the software fails tests down the production line, then I’ve initiated the feed-back loop of patching and re-testing that could have been avoided, and that is time, and that is money.

Now, unit testing suites will not catch every breaking change, nor every bug, but when breaking changes or bugs are discovered, new tests can be added to prevent them recurring. They will also catch many changes and bugs before getting to the testing teams or customers. This is the true value of a suite of unit tests, it becomes an asset to the business which reduces the cost of testing churn, and reduces the impact on software support staff.

I’d like to talk to the one ‘good’ excuse I’ve heard for why this value of unit testing doesn’t count…

“I authored that code, I know how to modify it without breaking it.”

This is the most delusional excuse of them all, because, even if you were the author of that piece of code – you are no longer the author! The author of that piece of code was an earlier you, that was focused on that piece of code, that wrote the code with recent context and understanding of how it would be used. The you that exists today, even if only days have passed, does not have the same focus on how that code works. Typically, not only days but months or years may have passed since you wrote that code. If you work in a team, almost certainly, that code has been consumed by other engineers that didn’t look at it’s source, possibly didn’t look at it’s documentation (if you wrote any), and likely used it in ways you never expected it to be used.

When you come back to modify code that you wrote months ago, your unit tests, and any that your co-workers have added, will be the last line of defense for the use cases under which that code has been used in production.

A dose of fear
Before I conclude this post, let me first give you a small dose of fear with regards to bugs.
Think about it, the software we write ends up in medical equipment, under the hood of a cars, in railway control systems, in vehicles which head out into space, in nuclear reactors, and now (thanks to Elon Musk) potentially in devices that attach directly to human brains.

We’ve seen cases of x-ray machines programmed to give too high a dose of radiation, or self-driving cars making their first mistakes. Every activity we take part in through our daily lives, involves software and therefore the potentially lethal consequences of software bugs.

Bugs are, terrifyingly, unavoidable.
The various forms of software testing that we have are our only protection from their potential evils.

I am no saint. I have written and even released code which did not have unit tests, and when I do write unit tests I sometimes do so after having written the code to be tested. It is not my intention to preach, but rather, I wanted to put into writing the various benefits of unit testing in particular.

I consider unit tests for my code to be an added value to my code base, and to anyone that will use what I’ve written. They are a sanity check and a safe guard. Unit tests can reduce long term costs, at the expense of a little time in the short term. Most importantly, I think that writing code without unit tests, while I am guilty of doing so myself, is a cavalier attitude which we should all strive to avoid.

Print Friendly, PDF & Email

2 thoughts on “The true value of unit testing.

  • I agree with you whole heartedly!

    Units test are most useful to mitigate breakage as the code is modified after release, because finding bugs, fixing them, and getting them out to your customers before those bugs cause havoc is time which is money. Releasing bugs affects your credibility and that’s even harder to fix afterwards.

    Unit tests serve as great documentation on how to use your APIs and force you to think like a tester, covering edge cases.

    You are 100% correct that even if you wrote it a week ago, you probably don’t remember all the nuances of the code and can modify it without any breakage. Besides that everyone has bad days..

    The biggest reason that XP has not been adopted is that businesses have not accepted all the studies that show unit testing pays for itself in the long term. Businesses are in it for the short term as products quickly become “obsolete” and want to quickly get them to market. As a result, sometimes building unit tests is very difficult as the initial design does not lend itself to testing since that was not a consideration up front.

    I worked for a company where I wrote tests that got broken, and the senior lead developer simply ignored the fact until 3 months later when a customer complained. Even afterwards he questioned the value of unit tests. It was hard to convince both management and the dev team that unit tests would pay for themselves especially since the product has been around a long time already. Unfortunately, I have lots of similar experiences. People in the industry since it’s inception tend to look at all the great products created without unit testing, and think it isn’t necessary.

    • Absolutely.
      I once worked on a maintenance team fixing bugs. One bug came in which I repaired, but my repair damaged another part of the app. Another engineer repaired his use case, which re-opened my issue. We bounced back and forth perhaps three times each fixing our own use case, unaware of the other before we realized. Had that company had a policy to put a test in place for bugs, neither of us could have reopened the others bug, because the test suite would have caught it.

      Engineers aren’t cheap, as you’ll be aware, the highest cost to any business is staff, and so if we each spend one-two hours per case, you could calculate the cost to the business. I’ve done this math before, and not wanting to mention numbers, an hour to write a unit test would have been cheaper. I feel as professionals, we should illustrate such cases to the business as it’s simply wasteful.

      Thanks for sharing your experiences!

Leave a Reply