To measure what percentage of code has been executed by a
test suite, one or more
coverage criteria are used. These are usually defined as rules or requirements, which a test suite must satisfy.
Basic coverage criteria There are a number of coverage criteria, but the main ones are: •
Function coveragehas each function (or
subroutine) in the program been called? •
Statement coveragehas each
statement in the program been executed? •
Edge coveragehas every
edge in the
control-flow graph been executed? •
Branch coveragehas each branch (also called the
DD-path) of each control structure (such as in
if and case statements) been executed? For example, given an
if statement, have both the
true and
false branches been executed? (This is a subset of edge coverage
.) •
Condition coveragehas each Boolean sub-expression evaluated both to true and false? (Also called predicate coverage.) For example, consider the following
C function: int foo (int x, int y) { int z = 0; if ((x > 0) && (y > 0)) { z = x; } return z; } Assume this function is a part of some bigger program and this program was run with some test suite. •
Function coverage will be satisfied if, during this execution, the function foo was called at least once. •
Statement coverage for this function will be satisfied if it was called for example as foo(1,1), because in this case, every line in the function would be executed—including z = x;. •
Branch coverage will be satisfied by tests calling foo(1,1) and foo(0,1) because, in the first case, both if conditions are met and z = x; is executed, while in the second case, the first condition, (x>0), is not satisfied, which prevents the execution of z = x;. •
Condition coverage will be satisfied with tests that call foo(1,0), foo(0,1), and foo(1,1). These are necessary because in the first case, (x>0) is evaluated to true, while in the second, it is evaluated to false. At the same time, the first case makes (y>0) false, the second case does not evaluate (y>0) (because of the lazy-evaluation of the Boolean operator), the third case makes it true. In programming languages that do not perform
short-circuit evaluation, condition coverage does not necessarily imply branch coverage. For example, consider the following
Pascal code fragment: if a and b then Condition coverage can be satisfied by two tests: • a=true, b=false • a=false, b=true However, this set of tests does not satisfy branch coverage since neither case will meet the if condition.
Fault injection may be necessary to ensure that all conditions and branches of
exception-handling code have adequate coverage during testing.
Modified condition/decision coverage A combination of function coverage and branch coverage is sometimes also called
decision coverage. This criterion requires that every
point of entry and exit in the program has been invoked at least once, and every decision in the program has taken on all possible outcomes at least once. In this context, the decision is a
Boolean expression comprising conditions and zero or more Boolean operators. This definition is not the same as branch coverage, however, the term
decision coverage is sometimes used as a synonym for it.
Condition/decision coverage requires that both decision and condition coverage be satisfied. However, for
safety-critical applications (such as
avionics software) it is often required that
modified condition/decision coverage (MC/DC) be satisfied. This criterion extends condition/decision criteria with requirements that each condition should affect the decision outcome independently. For example, consider the following code: if (a or b) and c then The condition/decision criteria will be satisfied by the following set of tests: However, the above tests set will not satisfy modified condition/decision coverage, since in the first test, the value of 'b' and in the second test the value of 'c' would not influence the output. So, the following test set is needed to satisfy MC/DC:
Multiple condition coverage This criterion requires that all combinations of conditions inside each decision are tested. For example, the code fragment from the previous section will require eight tests:
Parameter value coverage Parameter value coverage (PVC) requires that in a method taking parameters, all the common values for such parameters be considered. The idea is that all common possible values for a parameter are tested. For example, common values for a string are: 1)
null, 2) empty, 3) whitespace (space, tabs, newline), 4) valid string, 5) invalid string, 6) single-byte string, 7) double-byte string. It may also be appropriate to use very long strings. Failure to test each possible parameter value may result in a bug. Testing only one of these could result in 100% code coverage as each line is covered, but as only one of seven options are tested, there is only 14.2% PVC.
Other coverage criteria There are further coverage criteria, which are used less often: •
Linear Code Sequence and Jump (LCSAJ) coverage a.k.a.
JJ-Path coverage has every LCSAJ/JJ-path been executed? •
Path coverageHas every possible route through a given part of the code been executed? •
Entry/exit coverageHas every possible call and return of the function been executed? •
Loop coverageHas every possible loop been executed zero times, once, and more than once? •
State coverageHas each state in a
finite-state machine been reached and explored? •
Data-flow coverageHas each variable definition and its usage been reached and explored?
Safety-critical or
dependable applications are often required to demonstrate 100% of some form of test coverage. For example, the
ECSS-E-ST-40C standard demands 100% statement and decision coverage for two out of four different criticality levels; for the other ones, target coverage values are up to negotiation between supplier and customer. However, setting specific target values - and, in particular, 100% - has been criticized by practitioners for various reasons (cf.)
Martin Fowler writes: "I would be suspicious of anything like 100% - it would smell of someone writing tests to make the coverage numbers happy, but not thinking about what they are doing". Some of the coverage criteria above are connected. For instance, path coverage implies decision, statement and entry/exit coverage. Decision coverage implies statement coverage, because every statement is part of a branch. Full path coverage, of the type described above, is usually impractical or impossible. Any module with a succession of n decisions in it can have up to 2^n paths within it; loop constructs can result in an infinite number of paths. Many paths may also be infeasible, in that there is no input to the program under test that can cause that particular path to be executed. However, a general-purpose algorithm for identifying infeasible paths has been proven to be impossible (such an algorithm could be used to solve the
halting problem).
Basis path testing is for instance a method of achieving complete branch coverage without achieving complete path coverage. Methods for practical path coverage testing instead attempt to identify classes of code paths that differ only in the number of loop executions, and to achieve "basis path" coverage the tester must cover all the path classes. ==In practice==