Coverage type

Coverage, coverage type and coverage ratio

‘Coverage’ is a useful concept that helps testers with complex questions, such as:

Since it is impossible to test everything and we are therefore forced to test only a subset of all of the possibilities, what is the best subset?
What is the difference between ‘elementary testing’ and ‘thorough testing’? What does it mean concretely for the test cases we need for this?

Coverage has everything to do with the wish to find the most possible defects with the fewest possible test cases. Instead of simply testing ‘just any’ subset of options, we aim to compile a set of test cases that create the greatest possible change of finding the defects that are there.

We can never be certain that all defects have been found or even that 60% of all defects has been found. After all, we do not know how many defects there actually are. What we can demonstrate, is the coverage realised by the test. And this confers a certain level of confidence that the chance of any defects remaining in the tested system is small. In summary: the higher the coverage realised, the smaller the chance that any unknown defects remain in the system.

There are a lot of definitions relating to coverage. The definition below is a simple one, but it reflects the essence perfectly well:


Definition Coverage is the ratio between that which can be tested and that which is tested with the test set.

In other words, coverage says something about how many of ‘all possibilities that can be tested’ are actually tested. This means that there are 2 phenomena that together determine the coverage:

Which possibilities are distinguished?
This is termed coverage type. It indicates what type of possibilities are involved. These are subsequently elaborated into the required test situations. For instance, it is possible to look at the possible combinations of paths in an algorithm. But in that same algorithm, you can also look at the possibilities to cover one outcome or the other within each decision point. These are 2 different forms of coverage (i.e. ‘coverage types’).


Definition Coverage type is the form in which the covering of test situations deducible from the test basis, is expressed.

How many of these possibilities are tested?
This is called the coverage ratio and is generally expressed as a percentage. It shows which part of all of the alternatives have actually been tested. The number (percentage) does not have meaning until it is clear which possibilities are involved, i.e. when it is associated with the coverage type.


Definition Coverage ratio is the percentage of test situations, as defined by the coverage type, that is covered by the test.

The decision to ‘test more thoroughly’ concretely translates to a decision to achieve more extensive coverage. In principle, there are 3 options in this respect:

A more thorough coverage type
Multiple coverage types
A higher coverage ratio of a specific coverage type.

One type of coverage or more?

It should be clear by now that a statement like ‘I want to test with a 75% coverage ratio’ does not say much, for the following reasons:

What type of coverage are we talking about?
Why does 25% (and which) not require coverage? (In other words: Why not simply cover 100%?)

This section will discuss these issues in greater detail and explain

that we cannot talk about the coverage, but talk of many forms of coverage (i.e. ‘coverage types’)
that the choice for more thorough testing must be worked out primarily with the choice for another coverage type
that the use of a coverage type implies that one aims for a coverage ratio of 100%
that generally speaking, the various coverage types CANNOT be compared in terms of ‘X is better than Y’. Different coverage types tend to supplement rather than replace each other.

To explain this, we will use the following example about a system to order books via the Internet. It describes only a fraction of what such a system could do in reality, but it will suffice for our purposes.

Chart of the process of 'ordering books'


Example The chart below describes the process to order books in our example system. We have used an intuitive diagram technique with decision points (A, B, and C) and paths (1 through 7). In reality a lot is happening on the paths of the process, but this is not relevant to the example. It is mainly about the fact that the process includes different possibilities and the question which different aspects should or can be tested. The ordering process looks like this. The numbers and letters in parentheses refer to the decision points and paths in the diagram: After the general details of the ordering party are entered (1), it is decided (A) whether to continue with an existing order (2) or initiate a new order (3). An order may contain more than one book. After the required books are ordered, it is decided (B) whether to continue to payment (4) or change the order by cancelling ordered books (5), after which the order can continue. When determining the final price, it is checked if the customer is a ‘good customer’ (C) and is entitled to extra discount (6) or not (7). Figure: Chart of the process ‘ordering books’. Discussion on test cases and coverage ratio Say that the system component described above requires thorough testing. What does that mean exactly? Which test cases are required? A test case runs through the process to be tested from start to end, covering various decision points and going through different paths. For a thorough test, we must look for a set of test cases that responds to the objective in the best possible way: ‘If there is a defect in the process, then this set of test cases will find it.’ Now look at the following set of 2 test cases, which are described by a series of numbers that shows which paths are followed consecutively: TC-1 = 1 2 5 3 4 6 TC-2 = 1 3 4 7 Thanks to the loop in the chart, test case TC-1 covers virtually all of the paths. Only path 7 is not covered. This is why TC-2 has been added.So do these 2 test cases achieve a coverage ratio of 100%? If a coverage tool were used to measure the coverage, it would indeed report that 100% coverage is achieved. Still, many testers would say that these 2 test cases test far from everything… that many defects may exist in the system that would slip through these test cases. Are they right? Absolutely! For instance, look at this: Say that there is a defect in the system in path 5. When an ordered book is cancelled, a related discount must be cancelled as well – but the system incorrectly retains the discount. Is this defect found with the set of 2 test cases described above? No! While test case TC-1 does cover path 5 and will therefore receive an excessive discount, the latter then disappears when the test case continues through path 3 and starts a new order from scratch with discount=0. This defect would be found with a test case that continues with path 2 after completing path 5. In this case, the same order would be continued and the excessive discount would be visible on payment. In other words, this is a defect type that does not ‘occur in a path’ randomly, but occurs only in case of a special combination of 2 successive paths. So the 2 test cases TC-1 and TC-2 do have a coverage ratio of 100% in the sense of ‘covering all paths’, but they do not cover ‘all combinations of 2 successive paths’. For a set of test cases that does cover this, we speak of achieving ‘test depth level 2’.

Test depth level 2 = the certainty that all combinations of 2 successive paths are covered.

For ‘100% of test depth level 2’, path combinations 2-4 and 3-5 must also be covered. This can be achieved by adding the following test case:

TC-3 = 1 3 5 2 4 7

(Please note that this eliminates the need for test case TC-2.)

It is a very common phenomenon that paths affect each other, so that defects do not occur until paths are combined in certain ways. This influence may be functional (such as in the above example of excessive discount), but it may also be technical. An example of the latter is:
When cancelling an earlier discount in path 5, a buffer overflow occurs, overwriting the memory location of the variable ‘extra discount’. When a test case going through path 5 then runs through path 7 (where no extra discount is given), this defect will not be noticeable. However, if the test case would continue with path 6, a bizarre extra discount would be allocated or the application might even crash.

Naturally there may also be defects that do not occur unless a specific combination of 3 successive paths is followed. This defect type cannot be found for certain unless the set of test cases achieves test depth level 3. And so on… The higher the test level achieved, the greater the certainty that the more exceptional defects are found.

The concept of ‘test depth level’ and its application is explained in greater detail in the section on ‘Path coverage’.

So is this it: The higher the test depth level, the more thorough the test? The maximum test depth level is ‘infinite’, but this is clearly not practically feasible. But could one (theoretically) conclude that with test depth level ‘infinite’ one will find all defects? No! Not even that! The example below shows a defect that cannot be found with certainty even with a test set of test depth level ‘infinite’. And the defect described is not even farfetched:
A ‘good customer’ is given an extra discount. This is established in decision point C, where the system checks whether the customer has ordered a lot of books or for a high amount. More precisely:
IF (number of books > 8 OR amount ≥ €250) THEN extra discount

There are therefore 4 possible situations in which extra discount will or will not be given. These are shown below.

number of books

8 ; amount

≥

250 →

YES extra discount

(1)

YES

(2)

≤

≥

YES

(3)

≤

(4)

Say there is a defect in the system, so that you do not get the extra discount in situation (3). So, if you order few books but they are expensive, you will not get the discount – incorrectly.

The problem described above (situation 3) occurs in path 6, where you ARE supposed to get the extra discount. But there are 2 other situations also associated with path 6. Our test set of test depth level ‘infinite’ may have passed path 6 many times, but that does not guarantee that at least 1 test case has actually tried situation 3. It might accidentally be true, but we cannot be certain that situation 3 will be covered. Why not? Because the test set of test depth level ‘infinite’ is not designed for this purpose. The test set was designed to cover path combinations, but this defect has nothing to do with that. The defect would have been found with certainty if a test set was used with a different coverage type, i.e. a coverage that focuses on how a complex decision point handles the various options. E.g. by covering all possible combinations of all conditions. In the example above, this amounts to all 4 specified situations. Such a coverage type is called ‘multiple condition coverage’. Another coverage type is ‘decision coverage’, which covers only the 2 possible outcomes of the decision point (YES or NO extra discount). This is less thorough coverage than multiple condition coverage, which would not detect the above defect with certainty.

So what is best? Do we choose covering paths or covering decision points? Clearly doing both achieves the highest level of thoroughness. But say that this is not allowed (e.g. because it is too costly). Say that we have to choose, then what would be best? Or more concretely:
Which coverage type is better: ‘test depth level 2’ or ‘multiple condition coverage’?

This question may seem simple, but it isn’t. One coverage type could only be called ‘better’ with any practical use if it would find at least all of the defects found by the other coverage type plus some additional defects. But the example above demonstrated that multiple condition coverage finds a specific defect that is not found with test depth level 2 (i.e. the extra discount for few but expensive books). But the reverse is true as well (i.e. the discount that is given incorrectly in case of cancellation). So the best answer is:
We cannot unequivocally talk about ‘better’!
The various coverage types find different defects.


Tip If multiple coverage types can be used from which a choice has to be made, consult with the client. Analyse where the highest risks are located and select the coverage type that matches this area most adequately. Then make sure that the client is aware of the fact that there are risks in the area relating to the coverage type that was not selected. For instance, the tester and client must typically ask themselves the following for the earlier question whether to choose test depth level 2 or multiple condition coverage: Is it a process in which many different types of actions are executed whose consequences might affect each other? In this case, path coverage test depth level 2 would be a good choice. But in case of a process where a specific discount is given or not after every decision point, path combinations are not that interesting because the discounts do not have a mutual impact. However, the client might consider the possible situations in which the discount is given or not given extremely important. In this case, coverage type multiple condition coverage might be the wiser choice.

Tip

If multiple coverage types can be used from which a choice has to be made, consult with the client. Analyse where the highest risks are located and select the coverage type that matches this area most adequately. Then make sure that the client is aware of the fact that there are risks in the area relating to the coverage type that was not selected.

For instance, the tester and client must typically ask themselves the following for the earlier question whether to choose test depth level 2 or multiple condition coverage: Is it a process in which many different types of actions are executed whose consequences might affect each other? In this case, path coverage test depth level 2 would be a good choice. But in case of a process where a specific discount is given or not after every decision point, path combinations are not that interesting because the discounts do not have a mutual impact. However, the client might consider the possible situations in which the discount is given or not given extremely important. In this case, coverage type multiple condition coverage might be the wiser choice.

Substantiate the test strategy with test coverage types

That which we want to cover by tests is termed coverage type: the form in which test situations are deducible from the test basis. A coverage type focuses on achieving a specific coverage to detect specific types of defect.

Coverage types are found to be a difficult and time-consuming business. Some people would prefer not using any coverage types at all! It is true, some coverage types are difficult to apply and take a considerable amount of time. But who said testing was easy? Coverage types represent the only way to realise the agreed test strategy in a demonstrable way. If unsuitable coverage types are used or even none at all, it is impossible to demonstrate that the required test intensity is achieved. As a result, all of the choices that led to a suitable test strategy would be in vain, and it becomes very difficult to make a coherent statement as to whether test goals, risks, priorities or simply the desired test intensity have or have not been covered.

Coverage types may be divided into four basic coverage groups:

Process:
- Processes can be identified at several levels. There are algorithms of control flows, business processes. Coverage types like paths, statement coverage, and state transitions can be used to test (variations in) these processes.
Conditions:
- With almost every system, there are decision points consisting of conditions, where the system behaviour can go in different directions, depending on the outcome of such a decision point. Variations of these conditions and their outcomes can be tested using coverage types like decision coverage, modified condition/ decision coverage, and multiple condition coverage.
Data:
- Data is created and ends when it is removed. In between, the data is used by updating it or consulting it. This lifecycle of data can be tested, but also combinations of input data, as well as the attributes of input or output data. Some coverage types here are Boundary values, CRUD, Data flows, and Syntax.
Appearance:
- How a system operates, how it performs, what it’s appearance should be, is often described in non-functional requirements. Within this group we find coverage types like operational and load profiles, and presentation.

Coverage Types per coverage group

The table below gives a brief description of each coverage type per group. Where applicable, an indication is given how the depth of coverage can vary according to the coverage type.


Coverage group	Coverage type	Description
Process
	Control flow	Testing the program structure.
	Paths	Coverage of the variations in the process in terms of combinations of paths. A scheme of decision points and paths is required as a test basis.
	Rare events	Addressing events that happen very infrequently.
	Right paths / fault paths	Checking both the valid and invalid situations in every defined error situation. An invalid situation (faulty control steps in the process or algorithm that precede the processing) should lead to correct error handling, while a valid situation should be accepted by the system without error handling.
	State transitions	Verification of relationships between events, actions, activities, states and state transitions.
Conditions
	Decision points	Coverage of the various possibilities within a decision point with the purpose of arriving at the outcomes of TRUE and FALSE
	Semantics	Validation relationships between data.
Data
	Boundary values	A boundary value determines the transfer from one equivalence class to the other. Boundary value analysis tests the boundary value itself plus the value directly above it and directly below it.
	CRUD	Coverage of all the basic operations (Create, Read, Update, Delete) on all the entities.
	Data combinations	Testing of combinations of paramater values. The basis are Equivalence classes.
	Data flows	Verifying information of a data flow, which runs from actor to actor, from input to output.
	Domain testing	Coverage of a small number of values from a nearly infinite group of candidate values. Domain knowledge plays a very critical role while testing domain-specific work.
	Equivalence classes	The value range of a parameter is divided into classes in which different system behaviour takes place. The system is tested with at least 1 value from each class.
	Integrity rules	Checking the preconditions under which certain CRUD processes are or are not permitted.
	Syntax	Validation of attributes of input or output data.
Appearance
	Heuristics	Evaluation of (a number of) usability principles.
	Load profiles	Simulation of a realistic loading of the system in terms of volume of users and/or transactions.
	Operational profiles	Simulation of the realistic use of the system, by carrying out a statistically responsible sequence of transactions.
	Presentation	Testing the layout of input (screens) and output (lists, reports).

Variation in coverage types

For some coverage types, it is possible to variate the coverage thouroughness within the coverage type.

The table below gives several examples:


Coverage Type	Variation
Control flow	Statement coverage Decision coverage (branch testing/ arc testing) Paths
Paths	Test depth level N
State transitions	0-switch 1-switch 2-switch
Decision points	Condition coverage Decision coverage Condition/ decision coverage Modified condition/ decision coverage Multiple condition coverageCause Effect Graph
Semantics	See decision points and equivalence classes
Boundary values	Light (boundary value + one value) Normal (boundary value + two values)
Data	Right paths / Fault paths No data pairs One or some data pairs N-wise (extention of pairwise) All possible combinations
Integrity rules	See decision points and CRUD
Syntax	See individual test situations

Selection of Coverage types and Test design techniques

There exist many coverage types and test design techniques. For the sake of simplicity and practicality we will only highlight the most commonly used test design techniques and hence the application of the underlying coverage types.

To give you a practical overview we highlight the most commonly used coverage types and some test design techniques in which they can be applied.


Coverage type group	Test intensity: Light	Test intensity: Average	Test intensity: Thourough
Condition	Condition Decision Coverage – Elementary Comparison Test	Modified Condition Decision Coverage – Elementary Comparisson Test or Condition decision coverage – Decision Table Test	Multiple Condition Coverage Elementary Comparisson Test or Multiple Condition decision coverage – Decision Table Test
Data	One or some data pairs – Data Combination Test	Pairwise – Data Combination Test	N-wise or all combinations – Data Combination Test
Process	Statement coverage and Paths test depth level 1 – Process Cycle Test	Decision coverage and Paths test depth level 2 – Process Cycle Test	Paths test depth level 2 – Algorithms Test and Paths test depth level 3 – Process Cycle Test

We left out the coverage group “Appearance”. In the cases where that coverage group is applicable, the coverage type and thourougness depends too much on the specific interface and the result the tester wants to achieve.