A picture of me

Polyglot developer, trainer, and writer. Writing the The Guide to Real World TDD. Director of Training Services at 8th Light. Follow me on twitter.

Sign up to my newsletter so you never miss a new tip, and you'll get early access to the Guide to Real World TDD!

The Evidence For TDD

This blog was going to be something completely different.

Every week I do some trawling around the internet, looking for problems people are having that I can fix. This week I found a lot of comments like this:

Once a programming team has adopted a methodology it’s almost inevitable that a few members of the team, or maybe just one bully, will demand strict adherence and turn it into a religion.1

There are alot of software developers who feel this way. Agile is a “religion” or “cult” masquarading as serious software development. I really wanted to disagree, but the truth is that those of us in the Agile Development community do act like cultists. This is particular true of TDDites. Once a developer becomest test-infected they begin to treat any developer who doesn’t write tests like they are lazy, or an idiot. We stop asking ourselves why we test and start asking others questions like “don’t you want your code to work?” As if that’s helpful.

The Burden of Proof is On You

While we might think TDD is a universally accepted practice, most teams don’t do it. It’s not because they’re lazy or stupid, it’s largely because they don’t believe it’s helpful. Rather than shouting them down or dismissing them, the time has come to back up our arguments in favor of this strange practice with facts. To that end I’ve collected a bunch of papers that show actual evidence, as opposed to anecdotal evidence, of TDD working. I’d love to see what you have.

On the Effectiveness of Test-first Approach to Programming
Hakan Erdogmus, Maurizio Morisio, Member, IEEE Computer Society, and Marco Torchiano
NRC Institute for Information Technology; National Research Council Canada

This is an interesting paper where the authors did an experiment comparing a “Test After” approach with a “Test Driven” approach. The result?

Our main result is that Test-First programmers write more tests per unit of programming effort. In turn, a higher number of programmer tests lead to proportionally higher levels of productivity.

That’s right - they were MORE productive with tests, not less. Remember that next time somebody says they don’t have time for testing. To read the paper you’ll need to click “view deposited version.”

Test-Driven Development as a Defect-Reduction Practice
Laurie Williams, E. Michael Maximilien, Mladen Vouk
North Carolina State University, Department of Computer Science
IBM Corporation and North Carolina State University

NC State partnered with IBM to do a study on defect reduction with TDD. The results, from the abstract.

In this case study, we found that the code developed using a test-driven development practice showed, during functional verification and regression tests, approximately 40% fewer defects than a baseline prior product developed in a more traditional fashion. The productivity of the team was not impacted by the additional focus on producing automated test cases.

40% fewer defects and productivity was not impacted, but of course in a real world scenario they’d have to fix those defects - so I’d argue productivity improved.

Realizing quality improvement through test driven development: results and experiences of four industrial teams
Nachiappan Nagappan & E. Michael Maximilien & Thirumalesh Bhat & Laurie Williams

This paper, which also has E. Michael Maximilien and Laurie Williams as authors, had studies done with Microsoft and IBM. The results:

The results of the case studies indicate that the pre-release defect density of the four products decreased between 40% and 90% relative to similar projects that did not use the TDD practice. Subjectively, the teams experienced a 15–35% increase in initial development time after adopting TDD.

So some increase in development time but a drastic reduction in defects. I think it’s interesting to contrast with the first study that claimed an increase in productivity. This is just a guess, but students (the subjects of the first paper) can’t turn in incomplete work. It’s either done, or not finished, but professional programmers turn in incomplete work (code with bugs) all the time. Bug free code takes a little bit more time.

Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies
Thirumalesh Bhat, Nachiappan Nagappan
Center for Software Excellence, Microsoft Research

This is a study done entirely by Microsoft. Once again:

We observed a significant increase in quality of the code (greater than two times) for projects developed using TDD compared to similar projects developed in the same organization in a non-TDD fashion.

I haven’t found anything yet that says there isn’t an increase in quality or that the amount of time spent isn’t worthwhile, so remember that the next time your boss tells you to stop writing those stupid tests.

  1. http://typicalprogrammer.com/why-dont-software-development-methodologies-work/

Want to learn to write clean, working code fast?

Sign up to my newsletter to get early access to the Guide to Real World TDD and tips and tricks for making TDD effective in your codebase, regardless of language or platform.

If you liked this...