Quality risk analysis and Test strategy | DevOps

Where should the team focus their QA & testing activities? What should be the priorities of their tasks? To answer these questions the team needs to investigate the quality risks involved with the IT system they are creating or changing. When there is a high risk, more effort will be put in QA & testing, when there is a low risk, they will spend less effort.

A quality risk is a specific chance that the product fails in relation to the expected impact if this occurs. The chance of failure is determined by the chance of faults and the frequency of use. The impact is related to the operational use of the product.

The product to be tested is analyzed by means of a quality risk analysis with the aim of achieving a joint view, for all stakeholders, of the risky characteristics and parts of the product to be tested. This joint view on identified risks is input for determining the test strategy.

Test strategy is the allocation of quality measures to balance the investment in testing and to make an optimal distribution of effort over test varieties and test approaches to give insight in test coverage and test intensity. Often this is based on the quality risk levels and the pursued business value.

For more in depth information the building block ‘Creating a test strategy and test plan in high-performance IT delivery‘ is available.

There are many approaches to perform a quality risk analysis and to determine a test strategy. In the DevOps focused test strategy approach as described in this section, risk poker is used for achieving a joint view on the risks.

Risk poker

Risk poker is an extension of planning poker, which is often used for agile approaches, such as DevOps, in order to prepare an estimate relating to the size of the backlog items (for example user stories). Major differences are often encountered in respect to story points assigned by different participants. This is mostly caused by varying insights of the poker players concerning the required test efforts! A solution to achieve a better understanding regarding these test efforts, as well as ascertaining a more unambiguous size estimation of a user story, is through playing risk poker (more specifically: assessing the story risk). Risk poker is done just before planning poker. Risk poker, just like planning poker, requires a “whole-team” approach, which means that all team members will gain insight into, as well as achieve better understanding of, all the “ins” and “outs” of the various testing tasks. In short, just like planning poker, risk poker, besides assigning story points and risk points respectively, gains added value from conversation!

The DevOps-focused quality risk analysis and test strategy steps are (overview): 

1. Gather the DevOps team members.
2. List all backlog items (e.g. user stories, features) of the upcoming sprint.
3. Identify the relevant quality characteristics per item.
4. Analyze, using risk poker, the risk, which is a combination of the impact if a failure occurs and the chance of failure, for each combination of item and quality characteristic (this is called item risk).

Applying these first four steps results in a “Risk table”.

5. Determine the test intensity per combination of item and quality characteristic.
6. Allocate quality measures per item to each combination of test intensity and quality characteristic.

To determine the test strategy, the results of the fifth and sixth step are added to the “risk table”. The result is referred to as “test strategy table”.

In the following sections, the DevOps focused quality risk analysis and test strategy steps are described in detail.

Gather the DevOps team members

Just like planning poker, risk poker is a “whole-team” approach. However, there is a difference. The scrum master facilitates the risk poker session and will not participate in the poker game, but the product owner, contrary to planning poker, will participate. After all, the product owner represents all the stakeholders and will be able to provide valuable input, especially in the aspect of damage.

List all backlog items (e.g. user stories, features) of the current sprint 

For a common starting point, Ron Jeffries 3 C’s approach (Card, Conversation, Confirmation) [Jeffries 2001] is used to create a story. An actual (physical) Card (often a post-it note) with a user story on it. The Conversation segment provides additional explanations relating to the user story. This could be a visual annotation on the Card, or a reference to, for example, a use case and email exchange. The Confirmation (also called Acceptance) segment, usually on the backside of the Card, contains those elements needed to demonstrate the acceptance of the user story.

Listing all user stories and executing risk poker can be done at the same time as executing planning poker; usually during the sprint planning or at the end of a refinement session (during the previous sprint). The user stories (including the Conversation segment of the story card) should be “sprint ready” before starting risk poker (refer to step 4).

Identify the relevant quality characteristics per item  

In this step the team identifies other relevant quality attributes, besides functionality, for each item. A user story is often about functionality by nature, but maybe quality characteristics such as security, performance or usability are applicable as well. For a complete list of quality characteristics.

Analyze the possible impact and chance of failure for each combination of item and quality characteristic (is item risk) 

The quality risk analysis that is performed using risk poker results in item risks. 

Item risk is the chance of failure combined with the impact when a failure occurs. This is determined with risk poker and measured in risk points.

You can find many different variants of risk poker approaches on the internet, but the basics are all very similar. They are usually all based on “standard” product risk analysis approaches.

Risk poker is a fun and highly effective manner which can be used to assign risk points to user stories. There is no set rule as how to execute risk poker, but since most scrum teams work with poker cards anyway, it is easy to take the next step and use these same cards for risk poker as well. Below, you will find and explanation of the approach when using cards 1, 2 and 3 (where 3 = High, 2 = Medium, 1 = Low). Other sets are most certainly also possible. Instead of cards it is also possible to use a poker app.

The approach: 

1. Each team member gets/selects cards 1, 2 and 3.
2. The scrum master reads the item (often a user story) aloud at the start of the poker game. Of course, any team member can read the item out loud, but the scrum master often doesn’t play poker, which makes the scrum master the “perfect” facilitator for the poker game.
3. The scrum master requests the chance of failure estimates.
4. Each team member ascertains for themselves which card they believe represents the correct chance of a failure estimate and puts this card face down on the table.
5. Once everyone has made a selection, the scrum master will ask everyone to reveal their cards simultaneously.
6. If everyone has selected the same number of chance of failure points, the chance of failure estimate for this item can be considered complete.
7. If the chance of failure points differs, these differences will be discussed by the team (with a focus on the deviating values).
8. Repeat steps 4 to 7 until consensus has been achieved, after which the scrum master will write down the chance of failure points.*
9. Repeat steps 3 to 8 in order to get an estimate of the impact.* 
10. The result (the item risk points), which is the chance of failure multiplied by the impact, will be written down on the story card.

* Determine in advance after how many attempts to estimate an item, the attempts will be stopped, after which it will be set aside for further investigation.

In risk poker, the goal is not to get a perfectly accurate estimate. Assigning risk numbers can give a false sense of accuracy. That’s why many teams work with letters. For example, an A if the risk number is 9, a B if it is 6 or 4 and a C if 1, 2 or 3.

1, 2, 3	C	Low
4, 6	B	Medium
9	A	High

It is important to ensure that the entire team is working together on this in order to achieve mutual consensus. Furthermore, while playing risk poker, all team members often mention specific test focus points which are recorded on the Confirmation segment of the story card.

The result after applying the first four steps is a “risk table”. 

Determining the test intensity per combination of item and quality characteristic  

The symbols •••, •• and • are used to specify the test intensity. The more •••, the more intense the test. Three ••• must be specified for risk class A, two •• for risk class B, and one • for risk class C. Deviation from this rule is allowed, when there is a good reason.

There are several ways to proceed in this step. The DevOps IT delivery model is based on a whole-team approach. Therefore, a distinction between specific test levels is probably not made, in contrast to sequential and hybrid delivery models where it is much more obvious to use separate test levels. In DevOps, a distinction can be made between static testing, dynamic testing and other quality measures. In the example, the latter is worked out. If it is necessary to use test levels in DevOps, a column can be added per test level.

Allocation of quality measures per item to each combination of test intensity and quality characteristic 

“Quality is key”, is the credo in DevOps. But based on the item risk – which is ascertained prior to the sprint – it is possible for stories with a higher risk to be tested more extensively than stories with a lower risk. The risks and the ways in how to cover these are directly related to the acceptance criteria, as indicated on the Confirmation segment of the story card. Here, we can often find good situations as well as error situations and sometimes even a hint as to which test approach and coverage types should be used. In fact, a Confirmation completed in this way is a synonym for the Definition of Done for this specific user story. Furthermore, when thinking of ways to cover story risks do not just consider coverage-based and experience-based testing approaches, but also other quality measures, for example, verifying stories as being Independent, Negotiable, Valuable, Estimable, Small and Testable (INVEST).  And also think of approaches such as Pair Programming, Acceptance Test Driven Development (ATDD), Behavior Driven Development (BDD) and Test-Driven Development (TDD). An overview and explanation of quality measures can be found in Quality measures.

Above you can find an example after assigning the quality measure. In this table the dots are substituted by quality measures. Or, if it is desired that the test intensity remains visible, the dots can of course remain, and the quality measures are added. And, instead of using a table, you can just write the chosen quality measures on the story card.

For more in depth information continue to one of the following building blocks: