“100% Code Coverage”

There’s a nice exchange of ideas going on in the DZone ‘Javalobby’ based on a Nicolas Frankel blog entry called “100% code coverage!” that was posted recently on “A Java geek!“.  The original post is here.

Frankel’s assertion “code coverage and test are completely different things” is right on the money… as is his very lucid conclusion:

…assertions like ‘we must achieve 100% code coverage’ or ‘it’s a requirement’ cannot be taken as a general rule and is utter nonsense without some proper context.

…For teams to embrace quality is a worthy goal. It’s even more attainable if we set SMART objectives: 100% code coverage is only Measurable, which is why it’s better to forget it, the sooner the better, and focus on more business-oriented targets.

As is often the case, user comments made on the original article are priceless:

On the “A Java geek!” site, Thibault Delor comments,

“100% in code coverage is easy. 100% in test is impossible, there’s some thesis on the subject. Some methods allow to limits the risks, like formal methods, which are often expensive and so not always relevant.”

And on DZone, another great user comment:

“I worked on a USD$200 million dollar project, with 40+ developers and 2 million+ Lines Of Codes (excluding junits). In 3 years, we only managed 80% junit code coverage. Our selenium tests didn’t cover all UIs, but only UIs that are heavily used (and cannot live without), by customers. It’s very easy to achieve 100% code coverage on toy projects or projects that cost less that USD$1 million, but for a USD$200 million+ project, we find acceptance tests (on critical screens only) to be more useful.”

Unless your project has only a few authors, the feature set is sparse, and the problem space is trivial, it is usually impractical (and may be downright impossible) to test to the 100% level.  People are not perfect.  That includes the coders, the testers, and those that manage them.  Deal with it.  Commercial software development is not an academic exercise.  For business success, it must be the team’s working reality that some number of bugs will always exist in large-scale, complex software projects during some portion of it’s life cycle.  We want to minimize the number of bugs without setting an unrealistic expectation that a dynamic, complicated code base can be “bug free” (or at least for very long).

Of course, always strive for the highest quality solution.  That’s everyone’s job.  Code coverage is a tool in the toolbox – a mechanism that is both measurable and proven to improve product quality.  But, “100% code coverage” in real-world business is both meaningless and absurd.   ‘Smart, efficient testing’ must be the primary goal.

The reality is: unless budgets are infinite, complex products must go to market before they are “fully qualified”.  Spend your testing cycles wisely.