Tooling | DevOps

For DevOps teams to be able to efficiently deliver IT systems, end-to-end test automation that is fully integrated in the CI/CD process is essential and a pre-requisite for achieving good quality at speed. But many DevOps (or other Agile) teams struggle to achieve a good level of automation of their quality engineering activities. The World Quality Report shows as an important challenge: lack of guidance makes that many teams decide independently which tools they will use, which leads to a proliferation of tools in the organization. Also, the decision what to automate and what not, is not sufficiently considered.  More about ‘Test Automation‘ continue here.

In the DevOps culture, the teams strive to automate as many tasks and activities as possible. Therefore, tooling is a prerequisite for all activities in DevOps, so it is much broader than just test tooling.  

We will start this section with test tooling and then provide a more general overview of test tool capabilities and tool support needed in DevOps. In these sections we also give some examples of tools. However, the world of tooling evolves very rapidly.

Test tooling 

This section starts with the effects of test tooling, the types of test tooling are then listed, and the implementation of test tooling discussed.  

Effects of test tooling 

The use of any of these tools is aimed to produce an effect. It is useful to distinguish primary and derived effects.  

Test execution tools accelerate test execution and make it possible to execute more test cases, which means that the primary effect is efficiency. Regression tests can now be run every time code is merged with the main branch, which increases the stability of the IT system.

Cost reduction on the other hand, is always a derived effect. The remarkable thing is that the main financial benefits of using test tools is not within the test process itself: reducing test time benefits the business by adding business value earlier, improving quality benefits operations by reducing the number of incidents, and finding problems earlier benefits development by decreasing the costs for fixing them.

There is a choice in the derived effects: either reducing test execution time, increasing coverage in the same test execution time, or increasing the number of times the tests are executed. The exception in this category of test tools is a performance test tool, whose primary effect is the ability to execute a performance test; the derived effect is insight in performance and stability.

The other types of test tools have different derived effects. Test control tools have an effect on quality and progress control; test design tools save time; and test environment tools enable control over the preconditions to execute tests. The primary and derived effects of the most commonly used types of test tools are given below. Multiple effects to be achieved mean a choice should be made in test automation.

The effects of individual types of test tools can be increased by combining or integrating them. And even more benefit can be gained by combining or integrating them with other tools used in the application life cycle, such as tools used in the development process for requirements management, system design, development or deployment, and tools used in the operations process for change and issue management.

Types of test tooling

There are lots of different types of test tools, each with its own purpose. This also goes with different views on the classification of types or test tool. For instance, in his book, Maurice Siteur [Siteur, 2019] divides types of test tooling into two groups: Fault detection (Static and Dynamic) and Support (Management and Utilities). In this book, we classify test tools by stating the testing activities they support, the so-called “Test tool capabilities”.  

This does not mean that these individual activities are supported by a single test tool. Most test management tools feature a combination of testware management, anomaly management and reporting capabilities for example. In the below figure, a non-exhaustive list of needed capabilities and tool support for testing DevOps is given. Of course, there can be many more needed capabilities and possible tool support, depending on a specific project. The need for specific test tooling is also depended on the specific set of CI/CD test tools selected. Some functionality may be provided as part of available tooling, such as test management as part of a workflow, reporting as part for standard dashboards and (part of) test execution within the Integrated Development Environment (IDE).  

Implementation of test tooling 

To achieve the desired effects of using a test tool, we need to implement it. After implementation, the primary effect can be reached right away; the derived effects takelonger.

How to implement a test tool varies per type of test tool, but there are generic aspects that the implementation of any type of test tool should address. There is more to it than just installing the tool, as is visualized in the test tool implementation model.  

Most types of test tools have a strong relationship with the technology of the test object and thereby the development tools and technology used. This technology determines if a test tool can be used and if so, the amount of effort required to implement and maintain a usable solution. Another factor is the number of releases of the test object, which determines the frequency of use of the test tool as well as the frequency of having to maintain it. Some of the repetitive tool maintenance tasks can also be automated using tools, for example using Robotic Process Automation (RPA) tools.  

• Goals & Expectations  
  Setting goals in terms of business value, scope, results and timeframes is important, but dealing with expectations is equally important, they can differ substantially from what will be achieved. A business case is often drawn up for this. The predetermination of which goals should be achieved by automated testing is important to be able to evaluate if they are realistic and to later demonstrate that they have been achieved. The achievement of goals of automated tests must be demonstrated by keeping indicators. If necessary, the goals should be adjusted if they seem unrealistic. The expectations from automated tests are often high. Automatically implementing all test cases with one touch on a button sounds very appealing of course, but it is hardly realistic. Especially if it is expected that full automation of the test execution can be performed without impact on the process and the employees. The continuous evaluation of the expectations and, based on that, making the necessary adjustments is an important prerequisite for the successful implementation of automated tests.
• Commitment and preconditions
  The implementation starts with commitment and dealing with preconditions. The technical implementation deals with installing and setting up the test tool to create a usable and maintainable solution.
• Vision & policies  
  Tools for testing is just one topic, but there is more to think about. For instance, is there a vision and a policy about what the IT organization should achieve and how this will be achieved? And how does test automation fit in?
• Processes & roles
  Installing proper management of all products and resources is key to successful test automation. If the responsibility to maintain the products is not properly defined, the attention for maintenance diminishes and the automated tests become worthless over time. In addition, an important component of the installation of the processes and roles is determining the proprietors and underwriters of the products.
• Knowledge & skills
  A good technical implementation is not enough, people remain the critical success factor, and equipping personnel with the right knowledge and skills to use and maintain the test tool is essential. The implementation can become truly successful when using the test tool – and, of course, the necessary activities to be able to keep using it – has become an integral part of the testing process, or even better, the development process. In other words, when using the test tool has become self-evident.
• Automation framework & tools
  Test tools are resources, but implementation of a test tool for automated test implementation is more than just the installation of this test tool. Often the use of other resources (such as a test management tool or stubs) can deliver advantages. Therefore, tools need to be integrated in a test automation framework. It’s not just about the automated test cases, but also about reusable functions in the framework or, even broader, the tool suite.
• Changes and improvements
  After implementation, the test tool can be used, but one must keep adapting to changes to keep achieving the intended effects. The most obvious changes are changes in the test object that require maintenance in the test tool, but changes in organization or processes need adjustment too. Continuously looking for improvements on all levels helps us increase the effect of using test tools.

DevOps tooling 

In this section we will focus on the required capabilities and tool support for the people in the DevOps team. Having an integrated suite of tools that supports the teams activities will greatly benefit efficient and effective IT delivery. There is, however, much more to say about tooling. To prevent repetition, we will refer to the relevant sections.  


For example, there is tooling for a CI/CD pipeline, which is described in CI/CD;  


For continuous testing, test automation solutions, test orchestration, smart automated frameworks and “everything” as code automation see: Test automation;  
For TDM see test data management;  
For monitoring & control; 
For setting up a test environment see infrastructure. 

In this section, we will describe the capabilities and tools in more detail and refer to DevOps as a whole. This is a non-exhaustive list and there may be many more required capabilities and possible tool support, depending on a specific project:  

• Team Telemetry  
  Is about to understand the maturity of the team and how the team can improve.
• Live Site Telemetry  
  Is about measuring how the system runs, how the platform behaves and log management.
• Cognitive Monitoring  
  Is about how to automate and improve IT operations by applying artificial intelligence and machine learning to the log data.
• Security Monitoring  
  Is about monitoring security threats.
• User Telemetry  
  User sentiment and behavior are in DevOps the most important information for success and should be measured.
• Work
  Visualize and track work via either Kanban or scrum boards, depending on the flow of work. Choose an integrated tool for tracking work with traceability.
• Audit
  An important part of systems and controlling the work on these systems for regulations is the auditing and change management. Change management and approval flows should be incorporated in the automation and configuration release pipelines.
• Approval  
  Set approvals and control on the CI/CD pipeline and configuration management which control the deployment of a system.
• Service desk  
  A service desk with different levels of support should relate to the DevOps and should be fully monitored and stored.
• Collaboration  
  DevOps need collaboration tools to support and control the work. Choose a tool which will integrate with the regular development tools.
• Version Control  
  A version control system for DevOps supports capabilities like branching, merging and sees the differences between branches and workflows for reviewing and accepting changes.
• Build  
  Compile, validate and package the code is what a build system should do. Most activities are on static files, not a running system.
• Test  
  Testing on a system happens continuously. During build, provisioning, deployment and in production, a system should continuously be validated and tested on all needed scenarios as defined. Important for testing is reporting on execution history and the integration with tasks and features implemented by the team.
• Package Repository  
  A system is compiled into a package which can be deployed in the provisioned infrastructure. DevOps also use many open source packages during development which are either captured during deployment or incorporated in the package.
• Security  
  To embed security at the heart of DevOps there must be a focus on automation, validation and feedback loops from production.
• Configuration Management  
  Configuration management ensures consistency of all configuraton items with the environments involved. Be sure to separate the configuration of a system from the provisioning of that system (with infrastructure as code). This improves the immutability of the infrastructure and capability to make the infrastructure distributed.
• Infrastructure provisioning  
  Automated provisioning of the underlying infrastructure in a release pipeline is one of the most important activities of DevOps. Immutable infrastructure creates the infrastructure for the application repeatedly from the same file(s), only the configuration (see configuration management) may differ.
• Deployment package and container  
  The build (CI) process creates packages which are used for deployment in all environments, only configuration will differ. For container technology the same, and often in an easier way, concept is followed: the CI process creates the container with the application which is provisioned to all different platforms.
  
  

In the figure below you will find a non-exhaustive list of tools supporting the above-described capabilities.