Code coverage is “a percentage measure of the degree to which the source code of a program is executed when a particular test suite is run.” The instrument is old, older than most of the practices built on it: “code coverage was among the first methods invented for systematic software testing. The first published reference was by Miller and Maloney in Communications of the ACM, in 1963.”
The measurement is honest about what it measures, and the honesty is the problem. Google’s testing team states the limit in one sentence: “code coverage does not guarantee that the covered lines or branches have been tested correctly, it just guarantees that they have been executed by a test.” A line executed during a test run with no assertion attached to it counts exactly the same as a line whose every output is checked.
The idea
Coverage is an asymmetric signal, and treating it as symmetric is the whole failure. A low number is close to proof of a problem, since “a low code coverage number does guarantee that large areas of the product are going completely untested by automation on every single deployment.” A high number proves almost nothing, because “high coverage numbers are too easy to reach with low quality testing.” So the correct use of the tool inverts the intuition: “a lot of the value of code coverage data is to highlight not what’s covered, but what’s not covered.” The percentage is a byproduct. The list of uncovered lines is the output.
What “coverage” measures depends on the criterion
There is no single coverage number. “To measure what percentage of code has been executed by a test suite, one or more coverage criteria are used,” and the main ones ask different questions: function coverage asks “has each function (or subroutine) in the program been called?”; statement coverage asks “has each statement in the program been executed?”; edge coverage asks “has every edge in the control-flow graph been executed?”; branch coverage asks “has each branch … of each control structure (such as in if and case statements) been executed?”; and condition coverage asks “has each Boolean sub-expression evaluated both to true and false?”
The criteria form a partial order, so the same suite yields different percentages depending on what you asked for. “Path coverage implies decision, statement and entry/exit coverage. Decision coverage implies statement coverage, because every statement is part of a branch.” Which means a bare percentage is uninterpretable without its criterion: “67% branch coverage is more comprehensive than 67% statement coverage.”
One function, four different answers
Wikipedia’s worked case is a C function that sets
z = xonly when(x > 0) && (y > 0).Function coverage “will be satisfied if, during this execution, the function foo was called at least once.” Statement coverage is satisfied by a single call,
foo(1,1), “because in this case, every line in the function would be executed.” Branch coverage needs two,foo(1,1)andfoo(0,1), since the second fails the first condition and skips the assignment. Condition coverage needs three,foo(1,0),foo(0,1), andfoo(1,1), so that each Boolean sub-expression takes both values.One call gets you 100% statement coverage on this function. It also never exercises the false branch. The number did not lie; it answered a weaker question than the one you meant to ask.
Condition coverage and branch coverage are not nested in general either. “In programming languages that do not perform short-circuit evaluation, condition coverage does not necessarily imply branch coverage.” For Pascal’s if a and b then, condition coverage is satisfied by a=true, b=false and a=false, b=true, but “this set of tests does not satisfy branch coverage since neither case will meet the if condition.”
Above these sit the criteria used where being wrong is fatal. Decision coverage “requires that every point of entry and exit in the program has been invoked at least once, and every decision in the program has taken on all possible outcomes at least once.” For safety-critical applications “such as avionics software” it is often required that modified condition/decision coverage be satisfied, which “extends condition/decision criteria with requirements that each condition should affect the decision outcome independently.” Certification regimes then fix the target rather than leaving it to a team: “the ECSS-E-ST-40C standard demands 100% statement and decision coverage for two out of four different criticality levels; for the other ones, target coverage values are up to negotiation between supplier and customer.” Certain certifications go further still, since “DO-178B Level A” requires coverage at the assembly level when object code is not directly traceable to source statements.
There is also a dimension coverage instruments cannot see. Parameter value coverage “requires that in a method taking parameters, all the common values for such parameters be considered,” and Wikipedia lists seven common string values including null, empty, whitespace, valid, invalid, single-byte, and double-byte. Testing one of them “could result in 100% code coverage as each line is covered, but as only one of seven options are tested, there is only 14.2% PVC.”
Why 100% is not the goal, and why path coverage cannot be
Full path coverage is not merely expensive, it is generally out of reach. “Any module with a succession of n decisions in it can have up to 2^n paths within it; loop constructs can result in an infinite number of paths.” Worse, some of those paths are unreachable, and you cannot automatically tell which: “many paths may also be infeasible, in that there is no input to the program under test that can cause that particular path to be executed. However, a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the halting problem).” A coverage denominator that includes infeasible paths is a denominator no suite can ever fill.
That is the theoretical ceiling. The practical objection is different and sharper. Fowler frames the misuse plainly: “test coverage is a useful tool for finding untested parts of a codebase. Test coverage is of little use as a numeric statement of how good your tests are.” He quotes Brian Marick on the distinction that matters: “I expect a high level of coverage. Sometimes managers require one. There’s a subtle difference.” The mechanism is not subtle at all: “if you make a certain level of coverage a target, people will try to attain it. The trouble is that high coverage numbers are too easy to reach with low quality testing,” which yields “lots of tests looking for things that rarely go wrong distracting you from testing the things that really matter.”
His expectation for a thoughtfully tested codebase is a range, not a maximum: “if you are testing thoughtfully and well, I would expect a coverage percentage in the upper 80s or 90s. I would be suspicious of anything like 100% - it would smell of someone writing tests to make the coverage numbers happy, but not thinking about what they are doing.” Wikipedia’s criticism section cites exactly that line as the practitioner objection to fixed targets.
Google reaches the same conclusion from an organization with a lot of data. “A high code coverage percentage does not guarantee high quality in the test coverage. Focusing on getting the number as close as possible to 100% leads to a false sense of security. It could also be wasteful, burning machine cycles and creating technical debt from low-value tests that now need to be maintained.” They also name the incentive damage a target does even when the target is reasonable: “be mindful that engineers may start treating your target like a checkbox and avoid increasing coverage beyond the target, even if doing so would be prudent.” And on gating: “you should however be careful that it doesn’t turn into being treated as a checkbox that is required to be filled, as it can backfire (pressure to ‘hit the metric’ almost never yields the desired outcome).”
The bug class coverage structurally cannot see
Google splits production escapes into two kinds. Either “your tests did not cover a specific path of code, a test gap that is easy to identify with code coverage analysis,” or “your tests did not cover a specific edge case in an area that did have code coverage, which is difficult or impossible to catch with code coverage analysis.” Only the first kind is visible to the instrument. The second lives inside green lines. Their suggested remedy is a different instrument: “a better technique to assess whether you’re adequately exercising the lines your tests cover, and adequately asserting on failures, is mutation testing.”
What the number is actually good for
Fowler’s answer to the “so what is it for” question is narrow and correct: “it helps you find which bits of your code aren’t being tested. It’s worth running coverage tools every so often and looking at these bits of untested code. Do they worry you that they aren’t being tested?” He adds a footnote about audience that most coverage dashboards violate: “by ‘you’ here I mean the people writing the tests. Coverage is of little value to management since you need a technical background to understand whether the tests are good or whether the uncovered code is a problem.”
Google’s operational advice lands in the same place while being more concrete about process. “More important than the percentage of lines covered is human judgment over the actual lines of code (and behaviors) that aren’t being covered (analyzing the gaps in testing) and whether this risk is acceptable or not. What’s not covered is more meaningful than what is covered.” The delivery mechanism they recommend is code review, because “embedding code coverage into your code review process makes code reviews faster and easier,” and because when developers see “each covered line highlighted as part of the code review, they will make sure that the most important code is covered.” The reason that works is a fact coverage numbers erase: “not all code is equally important, for example testing debug log lines is often not as important.”
They do publish numbers, and the framing around them is as important as the values. “There is no ‘ideal code coverage number’ that universally applies to all products,” since the right level “should be a function of (a) business impact/criticality of the code; (b) how often you will need to touch/change the code; (c) how much longer you expect the code to live, its complexity, and domain variables.” With that caveat stated, “at Google we offer the general guidelines of 60% as ‘acceptable’, 75% as ‘commendable’ and 90% as ‘exemplary.’ However we like to stay away from broad top-down mandates and encourage every team to select the value that makes sense for their business needs.” The marginal-return shape explains where to spend: “we should not be obsessing on how to get from 90% code coverage to 95%. The gains of increasing code coverage beyond a certain point are logarithmic. But we should be taking concrete steps to get from 30% to 70%.”
Two operational details are easy to miss. First, the useful gate is on new code, not the whole repository: “while project wide goals above 90% are most likely not worth it, per-commit coverage goals of 99% are reasonable, and 90% is a good lower threshold.” Second, any mandate has to come with the means to satisfy it, since “any mandate to reach x% code coverage should be accompanied by infrastructure investments to make testing easy, such as integrating tools into the developer workflow.”
Sufficiency is a different question with a different measurement
The reason coverage cannot answer “are we testing enough” is that sufficiency is not a property of the code, it is a property of outcomes. Fowler proposes outcome tests instead: you are doing enough testing if “you rarely get bugs that escape into production” and “you are rarely hesitant to change some code for fear it will cause production bugs.” The second criterion is the interesting one, because it makes the test suite’s purpose confidence to change, which no line-execution count can observe.
He also allows the failure in the other direction. “Can you test too much? Sure you can. You are testing too much if you can remove tests while still having enough.” The symptom to watch is friction: “one sign you are testing too much is if your tests are slowing you down. If it seems like a simple change to code causes excessively long changes to tests, that’s a sign that there’s a problem with the tests,” possibly “duplication in your tests” rather than too many things tested. He is unpersuaded by slow suites as evidence of over-testing, since “you can always move slow tests to a later stage in your deployment pipeline, or even pull them out of the pipeline and run them periodically.”
One more caveat sits underneath all of this. Coverage is measured under conditions that are not production conditions, and the measurement itself perturbs the system. “Generally, test coverage tools incur computation and logging in addition to the actual program thereby slowing down the application, so typically this analysis is not done in production.” That slowdown is not neutral for concurrency bugs: “some race conditions or similar real time sensitive operations can be masked when run under test environments; though conversely, some of these defects may become easier to find as a result of the additional overhead of the testing code.”
How to read a coverage report
Ignore the percentage. Sort by uncovered lines, and ask Fowler’s question about each region: does it worry you that this is not tested? Then ask which criterion produced the report, because a statement-coverage green is a much weaker claim than a branch-coverage green on the same file. The number is for finding the list. The list is the artifact.
Related Notes
- Testing Strategies - what the tests coverage measures are supposed to be doing
- Code Review - where Google recommends coverage information be surfaced
- Continuous Integration - the pipeline stage a coverage gate would live in
- Technical Debt - low-value tests written to hit a number become maintenance load
- Refactoring - “rarely hesitant to change code” is the sufficiency test coverage cannot measure
- Incident Postmortems and Blameless Culture - production escapes are the outcome measure that matters
Sources
- “Code coverage,” Wikipedia. https://en.wikipedia.org/wiki/Code_coverage . Supports the definition of code coverage as a percentage measure of source executed by a test suite; the 1963 Miller and Maloney first published reference in Communications of the ACM; the use of coverage criteria and the definitions of function, statement, edge, branch, and condition coverage; the worked C
foo(x, y)example and which calls satisfy each criterion; the non-short-circuit Pascal example where condition coverage fails to imply branch coverage; the definition of decision coverage and of modified condition/decision coverage for safety-critical avionics software; the ECSS-E-ST-40C 100% statement and decision coverage requirement for two of four criticality levels; the DO-178B Level A assembly-level coverage requirement; parameter value coverage with the seven common string values and the 14.2% PVC illustration; the implication ordering among path, decision, statement, and entry/exit coverage; the 2^n path bound, infinite loop paths, infeasible paths, and the halting-problem impossibility of identifying them; the comparison that 67% branch coverage is more comprehensive than 67% statement coverage; the criticism of 100% targets citing Fowler; and the note that coverage tooling slows the application, is typically not run in production, and can mask or expose race conditions. - Martin Fowler, “Test Coverage,” martinfowler.com, 17 April 2012. https://martinfowler.com/bliki/TestCoverage.html . Supports coverage as a tool for finding untested code rather than a statement of test quality; the Brian Marick quotation about expecting versus requiring high coverage; the mechanism by which a coverage target is met with low-quality testing and distracting tests; the expectation of upper 80s or 90s and the suspicion of 100%; the two sufficiency criteria (rare production escapes, rare hesitance to change code); the discussion of testing too much including slow-changing tests and test duplication; the argument that slow suites can be moved later in the pipeline; the statement that coverage’s value is finding untested bits and the question to ask about them; and the footnote that coverage is of little value to management.
- Carlos Arguelles, Marko Ivanković, and Adam Bender, “Code Coverage Best Practices,” Google Testing Blog, 7 August 2020. https://testing.googleblog.com/2020/08/code-coverage-best-practices.html . Supports the statement that coverage guarantees execution rather than correct testing; that a high percentage does not guarantee quality and chasing 100% gives false security, wastes cycles, and creates technical debt from low-value tests; the two-kind split of production escapes and which kind coverage can detect; the recommendation of mutation testing; the guarantee implied by a low coverage number; the claim that the value lies in what is not covered; the absence of an ideal number and the three factors that should set it; the 60%/75%/90% acceptable/commendable/exemplary guidelines and the preference against top-down mandates; the logarithmic returns argument and the 30%-to-70% priority; the checkbox and metric-pressure warnings; the recommendation to embed coverage in code review and the observation that not all code is equally important; the per-commit 99% goal with 90% as a good lower threshold versus project-wide goals above 90%; and the requirement that any mandate be accompanied by infrastructure investment.