Coverage-based testing

The test strategy defines the intensity of (specific parts of) the testing of the test object. This test intensity is achieved by applying test approaches and test design techniques.  

The objective of an effective test strategy is therefore to realise the best achievable coverage at the right place. Coverage has everything to do with the wish to find the most possible defects with the fewest possible test cases.

In the following sections we will describe the coverage-based testing approach and the groups of coverage-based test design techniques. We will then share some examples of combining approaches and techniques. The experience-based testing approaches are described in the section “Experience-based testing“.  

Coverage-based testing 

Coverage-based testing aims to demonstrate a specific type of coverage of one or another aspect of an IT-system. This can be done by designing test situations and test cases with test design techniques.

In general, when applying a test design technique, different testers would end up designing the same test cases since the technique uses a specific way of working to derive test cases from the test basis.  

But what is coverage? Coverage is very subjective. We cannot talk about the coverage. What does an executive or other stakeholder mean when he/she asks you what the coverage of the test was? What information does he/she need or want? Possibly he/she wants to know how thorough some aspects of the test object have been covered. Maybe he/she want to know how many of all possible defects have actually been found by the tests.

A key word here is Coverage. A definition for coverage is hard to give. It basically deals with aspects of the test object that you would like to assess and the thoroughness with which you do that.

More important is the question if we are able to achieve 100% coverage. Well, we can never be certain that all defects have been found or even that 60% of all defects has been found. After all, we do not know how many defects there actually are. Furthermore we don’t know how accurate and complete the information was on which we based our test cases. Also if the tests we executed were based on a test strategy (and product risk analysis) we can never be sure whether our stakeholders made the right choices on what to cover. Testing everything is simply impossible, because it is impossible to define what ‘everything’ means.

Altough coverage is hard to define, it has a relation with the following two terms: 

• The parts of the test objects that we want to test
• Coverage thouroughness applied to each of those parts. 

Four groups of coverage-based testing 

There is a great variety of test design techniques. Based on a great number of sources we have found several dozens of test design techniques. Therefore, in practice it may be very difficult to decide which technique to use. Luckily you will never need all these test design techniques, applying somewhere between five and ten techniques is generally sufficient. To help make the choice which techniques to use in a specific situation, we have created four so-called coverage groups. All test design techniques can be assigned to one of these groups. It is wise to know at least one technique from each of these groups.

• Process  
  Processes can be identified at several levels. There are algorithms of control flows, business processes.
• Conditions  
  With almost every system, there are decision points consisting of conditions, where the system behaviour can go in different directions, depending on the outcome of such a decision point. 
• Data  
  Data is created and ends when it is removed. In between, the data is used by updating it or consulting it. This lifecycle of data can be tested, but also combinations of input data, as well as the attributes of input or output data.
• Appearance  
  How a system operates, how it performs, what it’s appearance should be, is often described in non-functional requirements.

The next sections give an overview of the test design techniques for each of the groups. Click on the test design technique for a more in-depth explanation.

Process-oriented test design

In process-oriented test design we distinguish the following test design techniques and coverage types:

• State transition testing (with n-switch coverage)
• Path testing (with test depth level-n coverage) which is used in the test design techniques Process cycle testing and Algorithm testing
• Statement testing (related to control flow testing)

Condition-oriented test design

In condition-oriented test design we distinguish:

• Control flow testing
• Semantic testing
• Elementary Comparison testing
• Decision table testing

For each of these test design techniques the same coverage types can be applied: Condition Decision Coverage (CDC), Modified Condition Decision Coverage (MCDC) and Multiple Condition Coverage (MCC).

Data-oriented test design

In data-oriented testing we distinguish:

• Equivalence partitioning
• Boundary Value Analysis
• Data combination testing (e.g. using pair-wise or n-wise testing)
• Data cycle testing (using CRUD)
• Data flow testing

Appearance-oriented test design

In appearance-oriented testing we distinguish techniques and approaches to testing based on various quality characteristics:

• Syntactic Testing (functionality)
• Load models (performance)
• A/B testing & Multivariate testing (usability)
• Abuse & Misuse cases (security)
• Other non-functional Testing techniques

Summarised

• It is not possible to test everything within the confines of the preconditions of time and costs defined in the assignment. Choices will have to be made as to the lengths one wishes to go to in testing.
• A test strategy is used to create an overview of what will be tested and how thorough, such that the aspects to be tested are covered as adequately as possible.
• The decisions concerning thorough and less thorough testing are translated to concrete statements about the targeted coverage.
• Depending on the available test basis, among other things, appropriate coverage types are selected to achieve said coverage.

There are many techniques, which one to use? Continue here.

Combining coverage- and experience-based testing approaches

Based on the test strategy, the team decides whether to use coverage-based testing or experience-based testing as the main approach in a specific situation.

However, we strongly advise to always have some level of combination of the two approaches. If, for example, the team chooses to use experience-based testing as the main approach, the tester that encounters a boundary, should still do a boundary value analysis for that boundary. On the other hand, if the team chooses to use coverage-based testing as the main approach, for example because they will automate all tests, meaning they will script them, the team should also do some experience-based testing; for example an exploratory test of a team member together with a user.

One reason for such an exploratory test is that a testing tool may not notice an obvious fault. For example, if there are white letters on a white background, the tool will still pass the test that checks if the right letters are displayed, whereas people will not see anything and raise an anomaly report. So, keep in mind to always combine experience-based and coverage-based testing to some extent.