Start typing keywords to search the site. Press enter to submit.
Generative AI
Cloud
Testing
Artificial intelligence
Security
Quite often, test managers are expected to answer such questions as:
Answering these types of questions with well-founded, factually based answers is not easy. Most questions can be answered with reference to the periodic reports. Such reports can only be created on the basis of correctly recorded relevant data, which is converted into information and then used to answer the above-mentioned questions.
The definition, maintenance and use of metrics is important to the test process because it enables the test manager an answer, supported by facts, to questions like:
A structured approach to realize a set of test metrics is using the Goal-Question Metric (GQM) method. In addition to describing the GQM method, TMAP gives instructions to set up a practical test metrics starter set. It also provides a checklist that can be useful to make pronouncements on the quality of the object to be tested and the quality of the test process.
For the test process, metrics on the quality of the test object and the progress and quality of the test process are extremely important. They are used to manage the test process, justify the test recommendations, and compare systems or test processes. Metrics are also important to improve the test process through assessing the consequences of certain improvement measures.
Metrics on the quality of the test object and the progress of the test process are of great importance to the test process. They are used to manage the test process, to substantiate test advice and also to compare systems or test processes with each other. Metrics are important for improving the test process, in assessing the consequences of particular improvement measures by comparing data before and after the measures were adopted.
When metrics are being collected, the test manager should take the following issues into account:
Below is an indication of what test managers embarking on a “metrics programme” should start with. The metrics set described is a starting set that can be used in practice with little cost and effort. “Reporting” lists a number of more specific test statistics and progress reports.
These elementary metrics (hours, documents and defects) can be used to assess the productivity of the test process. Note that this productivity should be seen in relation to the required effort and size of the test project. Example: in the first ten hours of testing we may find more defects per hour than in 400 hours of further testing, simply because the first defects are found more quickly than the last ones.
The following metrics regarding productivity can be derived from this elementary set:
If the number of function points or the number of ‘kilo lines of code’ (KLOC) of the object under test is known, the following numbers can be calculated:
For the test basis we can establish the following metrics:
When it is known how many defects occur in production during the first three months, the following metric can be determined:
The DDP can be calculated both per test level and overall. The DDP per test level is calculated by dividing the number of found defects from the relevant test level by the sum of this number of found defects and the number of found defects from the subsequent test level(s) and/or the first three months of production. The overall DDP is calculated by dividing the total number of found defects (from all the test levels together) by the sum of this number of found defects and the found defects from the first three months of production.
Some causes of a high or low DDP may be:
By recording the above-mentioned metrics, supplemented here and there with particular items, we arrive at the following list of metrics.
In the following (non-exhaustive) list of metrics, a number of commonly used metrics are mentioned, which can be used as indicators for pronouncing on the quality of the object under test or for measuring the quality of the test process and comparing against the organisation’s standard. All the indicators can of course also be used in the report to the client:
Savings of the test Indicates how much has been saved by carrying out the test. In other words, what would the losses have amounted to if the test had not been carried out?
To summarise, a test manager should record a number of items in order to be able to pass well-founded judgement on the quality of the object under test as well as on the quality of the test process itself. The following sections describe a structured approach for arriving at a set of test metrics.