GQM

Improvement of the application being delivered

TMAP is all about improving products, processes and people. The Goal Question Metric (GQM) is about improving products.

The improvement of the application being delivered is strongly related to the metrics as defined in Metrics. You could use the GQM approach to identify Goals, Questions and Metrics [Basili 1994]. When using GQM you can distinguish between two types of goals:

1. Knowledge objectives ⇒ Where do we stand right now? These objectives are expressed in words such as evaluate, predict, or monitor. For example, “Evaluate how many hours are actually spent on re-testing” or “Monitor the test coverage”. The goal here is to gain insight. 
2. Improvement objectives ⇒ Where do we want to go? These objectives are expressed in words such as increase, decrease, improve, or achieve. Setting such goals suggests that we know there are shortcomings in the present test process or the present environment and that we want to improve these. 
   
   

An example of such an improvement goal is obtaining a 20% saving on the number of testing hours at a constant test coverage within a period of 18 months. 

In order to ascertain this, the following two knowledge goals should be aimed at:

• Insight into the total number of testing hours per project.
• Insight into the achieved test coverage per project.
  
  

It is important to investigate whether the goals and the (test) maturity of the organisation match. It is pointless to aim at achieving a certain test coverage if the necessary resources (knowledge, time and tools) are not available. 

We will explain this using an example. 

Step 1 – Goal

First, we formulate the goal. Let’s assume the knowledge goal is: “Provide insight into the quality level prior to release”.

Step 2 – Questions

For each goal, several questions have to be asked. The questions are formulated in such a way that they act as a specification of a metric. It can also be asked, for each question, who is responsible for the test metrics supplied. From the above goal, various questions can be derived. We will limit the number of questions in this example to three. 

“Are the most important risks covered/found early in the DevOps QA and testing activities?””Are the most important risks covered/found in the DevOps QA and testing activities?””What is the stakeholder’s perception of the covered/found risks?”

Step 3 – From questions to metrics

The next step is deriving relevant metrics from these questions. You could use some of the already defined metrics in, or even better, use the metrics for which data is already available. Possible metrics are:

• Percentage of critical anomalies found early in test execution (effectiveness)
  Ratio between the number of critical anomalies detected in the first quarter of the test execution lead time and the total number of critical anomalies found in the total test execution lead time.
• Percentage of critical tests run early in test execution (effectiveness)
  Ratio between the number of critical tests run in the first quarter of the test execution lead time and the total number of critical tests run in the total test execution lead time.
• Percentage of identified risks covered by (executed) tests (effectiveness)
  Ratio between identified risks and the total number of (executed) tests in relation to these risks.
• Average cost per risk item covered during testing (efficiency)
  Total test costs divided by the total number of risk items.
• Stakeholder’s perception of accepted risks prior to release and after it has been released into production (customer satisfaction)
• Survey of risk perception prior to release to be completed by the stakeholders, and the same survey after three months of production. Make a comparison between both survey results.
  

Step 4 – Data collection and analysis

During the development process a variety of data is collected. One way of keeping things simple is to use forms/templates (if possible in electronic form). The data should be complete and easy to interpret. In the design of these forms, attention should be paid to the following points: 

• Which metrics are collected on the same form.
• Validation: how easy is it to check whether the data is complete and correct.
• Traceability: forms supplied with the date, project ID, configuration management data, data collector, etc. Take into consideration that it is sometimes necessary to preserve this data for a long time. 
• Possibility of electronic processing.
  
  

As soon as the data is collected, it should be analyzed. At this point it is still possible to make corrections. Waiting too long decreases the chance of restoring the data. Bear in mind possibilities, for example, of booking time with the incorrect activity code. 

Step 5 – Presentation and distribution of the measurement data

The collected measurements are used both in the test reports on the quality of the product under test and in those on the test process. Proper feedback is also of importance to the motivation of those involved and the validation of the measured data. 

Step 6 – Relating the measurement data to the questions and goals

This last step is used to investigate to what extent the indicators (answers to the questions) offer sufficient insight into the matter of whether the goals have been achieved. 

This situation may be the starting point for a new GQM cycle. In this way, we are continually improving the test process.