There are many techniques, which one to use?

There are a great number of approaches and techniques, which one(s) should you use? You can choose static testing or dynamic testing or (often) both. But what should you do in your specific situation?  

The decision which test design technique can best be used, can be very well prepared. It is never a random or ad hoc choice. The team has performed a quality risk analysis and now has insight into the areas with higher and lower risk. Usually, there is a need for higher coverage when there is a higher risk, and less coverage where there is a lower risk. This is often described in a test strategy.  

Definition A test strategy describes the allocation of quality measures to balance the investment in testing and to make an optimal distribution of effort over test varieties and test approaches to give insight in test coverage and test intensity. Often this is based on the quality risk levels and the pursued business value.

A team could create a test intensity table once to guide the team members in the choice for test approaches and test design techniques.

The final choice depends on many variables, for example

• The pursued business value. For example, when this is crucial to the whole organization, the team will select a thorough technique whereas when the business value is relatively low, the testing will be not very intense.  
  Furthermore, with a higher business value there will usually be more budget available for testing. But don’t forget that with a very low business value (which would lead to believe that there is little budget for testing) there still may be high quality risks. Often, when the risk is high and the value is low, it is better not to develop the feature in the first place (using the saying “no risk – no test – no development”, see below).  
• The kind of testing challenge at hand often relates to the coverage group from which a test design technique should be selected (for example: if the challenge relates to testing data, a technique from the coverage group “data” should be selected, such as boundary value analysis, but if the challenge is related to a system with various states, then state transition testing is a good choice).  
• The quality risk level. With a high risk, a technique that gives higher coverage should be selected, with a low risk a technique with lower coverage will suffice. Or with low risks, the team may choose to only apply an experience-based approach such as exploratory testing.  
• The available test basis. Most test design techniques have specific requirements for the information needed to derive test cases, if the test basis does not contain that relevant information, another technique should be chosen, or the test basis should be extended.  
• The available time. If there is enough time to do proper test design upfront, a test design technique is applied, if there is hardly any time available, an experience-based approach such as exploratory testing can be applied.  
• The skill level of the team member(s) involved. In many cases there is more than one test design technique that could be applied. It is wise to select a technique that team managers are skilled at. Otherwise training or coaching should be arranged.  
• And in your specific situation there may also be other considerations possible to choose for one or more test approaches and test design techniques.  

And keep in mind that coverage-based testing should always be combined with experience-based testing, in some way or another. 

No risk – no test – no development 

Often, people involved in risk-based testing use the phrase “No risk – no test”. This sentence is incomplete. It is better to use the statement: “No risk – no test – no development”. Because when there is no risk that means that if the feature fails, nobody has a problem and thus nobody really needs it. It is therefore a waste of money to develop it. Especially since you also run the risk that there are faults that cause more damage than the total business value of the feature.  

More information can be found in the building blocks “test strategy table” and “test intensity table“.

Coverage type

A more easily definable term closely related to coverage is the coverage type, the form in which test situations are deducible from the test basis. A coverage type focuses on achieving a specific coverage to detect specific types of defect (e.g. in the interfaces, the input checks or the processing), such defects are detected more effectively then by specifying ad hoc test cases. One coverage type could only be called ‘better’ with any practical use if it would find at least all of the defects found by the other coverage type plus some additional defects.

The decision to ‘test more thoroughly’ concretely translates to a decision to achieve more extensive coverage. In principle, there are 3 options in this respect:

• A more thorough coverage type
• Multiple coverage types
• A higher variation of thoroughness within a specific coverage type.

A term to be introduced here in relation to coverage is coverage thoroughness.

What exactly is coverage thouroughness?

In the test strategy is decided what coverage thouroughness is to be acheived in the test. This means that executives and other stakeholder most likely expect information about the thoroughness of your test. But what do they mean? They probably mean one of the four aspects related to thoroughness:

• Did you cover everything?
• How thorough was the chosen coverage type?
• Were multiple coverage types applied?
• How high was the variation of thoroughness within a specific coverage type?
  

So, although a fascinating subject, it’s also a complex matter. There is no black and white here. However, we can state that testing everything is impossible. How more thorough is a certain coverage type compared to another (e.g. pairwise testing versus modified condition/decision testing)? How more thorough is one variation within a coverage type compared with another variation within that same coverage type (e.g. modified condition/decision testing versus modified condition coverage)? How many additional defects are to be found? 

Example case: “The magic boat ride in QualityLand” 

We describe various test design techniques, understanding how these work often goes best by using an example. To demonstrate how different test design techniques can be applied to the same functionality, we will use one example and for each technique create the test cases for this example.  

The example case is about an amusement park. In this amusement park, called “QualityLand”, there are many different attractions. One of them is the “magic boat ride”. For a group of people to be allowed on this magic boat ride, there are a few rules that the group must comply with:  

Example Everyone in the group must be at least 120 cm tall and the total weight of the people in the group must be at least 100 kg and no more than 300 kg.

The process flow description for this example is shown in below figure. The Process flow for allowing access to the magic boat ride. 

For each of the four coverage groups, we list the test design technique(s) in that group that are most relevant in a DevOps context. Per technique we mention the coverage that can be achieved and we show for this example what test cases the technique results in. 

Coverage group	Coverage typein a DevOps context
Process-oriented testing	Path coverage	Continue here
Condition-oriented test design	Condition coverage (CC)Decision coverage (DC)Condition decision coverage (CDC)Modified condition decision coverage (MCDC)Multiple condition coverage (MCC)	Continue here
Data-oriented testing	Equivalence partitioning (EP)Boundary Value Analysis (BVA)	Continue here