Reporting & Alerting

Aim

Creating reports to provide insight into both the quality of the test object and the progress and quality of the separate test levels and the total test process. These reports ensure that the client and other stakeholders can manage effectively based on the progress of the test process.

Method of operation

During the test project, the test manager creates various reports. In the master test plan, the chapter “Management” specifies the type and frequency of reporting. Periodical reports are made on the quality of the test object and the progress and quality of the test process. In addition to the periodical reports, the client or other stakeholders may request ad hoc reports. The most common example is the risk report to map the possible consequences of a threat or risk to the test process and propose countermeasures. Furthermore, a sudden request for an additional progress report may be made, e.g. as most up-to-date input for a project board or project management meeting. A release advice and final report are compiled at the end of the test process. 

All this information provides the client, project manager and other stakeholders with insight into the degree to which:

• the intended result has been achieved
• the risks during commissioning are known and as small as possible within the framework of the defined preconditions 
• this is done within the defined budget and proposed (lead) time.

In other words, these are the BDTM aspects of Result, Risks, Time and Costs. Providing insight means that the report must match the perception of its recipients. 

The reports are based on the data as recorded that conforms with the test process management. 

Test process management focuses on managing the test process in terms of progress and quality and providing insight into the quality of the test object. To this end, the following data must be identified, registered, administrated, stored and interpreted: 

• Progress and use of budget and time
• Quality indicators
• Test statistics.

This information represents the basis for control and reporting by the test manager. The test manager of each test level is responsible for managing his own test process. The overall test manager ensures that this is done correctly. To this end, the test manager checks, randomly if necessary, the separate test levels (or orders checks) both for progress and quality of the test process. The latter can be achieved by e.g. asking the method test expert to check the traceability from test basis to test scripts to test results. The enforcement of instructions, approach and procedures can also be verified. If required, a separate report can be made on this aspect. 

The progress data are derived from the data and reports of the individual test levels and by recording the data on the overall activities and products. Maintaining the same types of data for all test levels is therefore vital to the creation of overall statistics. 

According to the BDTM view, reporting takes place on the four aspects of Result, Risks, Time and Costs.

• Result 
  • The outcome of the tests executed at the level of characteristic/object part
  • The result in terms of obtained/not-obtained test goals (business processes, user requirements, etc.)
  • Any trend analyses
• Risk 
  • Detection of parts that are being tested more superficially (or not at all) than the risk estimate indicates, thus presenting a higher risk 
  • Detected (test) project risks
• Time + Costs 
  • Progress of testing (in activities, products, hours spent and, optionally, money, dates)
  • Indication of when the testing will be completed.

Reporting on risks and results takes place at the level of test goals, as agreed with the client and other stakeholders. The risk tables of the product risk analysis are maintained with this aim. It is up to the test manager to translate test results on characteristics/object parts effectively, and on the basis of the tables, to this level. 

Reporting can take place in various ways, to various target groups and at various times. The most important forms of reporting are: 

• Progress and quality reports 
  Information and advice on progress (and, optionally, quality) of the test process and on quality/risks of the test object, based on the four BDTM aspects. 
  Frequency: periodically, preferably weekly 
• Risk report 
  With certain (project) risks, the test manager can, either upon request or at his own initiative, report on risk, the consequences for the test process and possible measures for dealing with the risk. In the Prince2 project management method these are known as ‘exception reports’. 
  Frequency: ad hoc 
• Release advice 
  Information and advice on quality/risks of the test object + formally established release advice. 
  Frequency: towards the end of the test execution, before the decision has to be taken on release 
• Final report 
  Evaluation of the test process and test object, looking back from the original plan.
  Frequency: once, at the end of the test process. 

A more detailed description of these reports is found in the wiki’s by clicking on the report.

The test manager will determine, for each of these forms of report, to whom they should be sent, whether for approval or for information, with what content and degree of detail and with what frequency. In the activity, “Understanding the assignment” the test manager has already looked at which parties should, or wish to, receive reports. In consultation with the client, that is now determined in more detail. As an aid in overseeing who should receive which report, a matrix can be set up of report forms and target groups. 

Products 

Reports (progress report, risk report, release advice, final report)  
Experience data  
Cost/benefit analysis.

Techniques 

Checklist “Test process evaluation” (see ‘Checklists‘)
Metrics.

Tools 

Defect management tool  
Testware  tool  
Workflow tool  
Planning and progress monitoring tool.