Test automation | DevOps

The demand for continuous feedback has created a renewed focus on test automation, for example due to developments within IT technology as a whole and commercial aspects. The popularity of agile approaches, from Scrum to DevOps and agile at scale, is one of the main drivers of this development. There is a need for continuous improvement and “quality at speed,” which consists of three pillars: shift-left, continuous testing, and continuous feedback loop. 

Automate everything you can is one of the six DevOps principles. Nevertheless, we still see many teams struggle with effectively applying automation in testing. One of the reasons is that it is often only focused on automating the execution of test scripts instead of the broader scope of automation in testing. For example, when people talk about test automation, people tend to directly select a test automation tool and try to use this to automate their current manual test script. They forget to look at things like: which people will work with it, what problem they are solving with automation, what is the value of automation for the business, and what is the impact on the current process. Implementing the tool becomes the goal on its own instead of improving the quality of products, processes and people. 

Automation in testing is one of the main opportunities to meet the need for speed. Still, it also requires a structured approach with accelerators to realize such a vision effectively. This can be achieved by addressing challenges like business alignment and automation maintenance.  

In this chapter, we discuss which quality engineering & testing activities should be automated and which test varieties should be organized (supporting shift-left). We describe the following subjects: continuous testing, current test automation solutions, test orchestration (supporting continuous feedback loop), smart frameworks, and “everything” as code automation.

For more in depth information about organizing test automation read the building block “Organizing automation of quality engineering“.

Treat it as a product 

To maximize the value of automation in testing, it is essential to treat your test script code the same way you treat production code. Make the scope explicit and consider future changes to this scope. The scope can cover systems, test varieties, or teams using automation. Determine the main, business-oriented, goal the automation contributes to and identify how progress towards this goal is measured. For example, when your business goal is to shorten your time to market, your related automation goal can be to improve efficiency, a way to measure this is the lead time from the refinement of a user story until having it available for the users in the production environment. In other words, we are not measuring the automation level or the number of automated scripts but we focus on the end goal and check whether automation contributes to that. 

When you have set the pursued business value and the objectives of the automation, you can focus on the organizational and technical needs. Think of a proof of concept (PoC) and tool selection, ownership, infrastructure setup, team members’ training, and budgeting. 

People 

People, not tools, are critical to successfully implementing automation in testing. A lasting business value should be achieved with an effective and sustainable solution, this is not something tools can do without human effort. Therefore, stakeholders must know about automation and the importance of automation. Roles and responsibilities must be clear, and people are aware of their roles. Different expertise need to work together; critical skills required are: 

• Testing; 
• Automation; 
• Infrastructure. 
  

Often this is not something that one person can do on their own, collaboration is key.  

How to determine which tests should be automated and which test variety should be selected 

Team members often struggle to determine what to test manually and what to check with automated test execution. The testing pyramid [Cohn 2009] can be used to decide on which level to automate. The basic message of this testing pyramid is that feedback should be collected as early as possible with automated checks as much as possible in the code and not via the user interface. And when deciding on test varieties, many testers find great support in the testing quadrants [Marick 2003, Crispin 2008]. Both models can be adjusted to a specific DevOps situation. The adjusted pyramid or adjusted testing quadrants are often described in the way of working or Definition of Done in DevOps or – at a higher level – are included in a Quality & Test policy. As an example, we have added our adjusted models. 

Continuous testing 

A specific subject in a test strategy for DevOps is the automation of checks. Since DevOps usually coincides with continuous delivery, most checking should be automatically performed during the process. Most test execution tasks should be automatically performed during the continuous integration and continuous delivery or continuous deployment process. Therefore, the term continuous testing is often used.

Continuous testing is more than just automating tests. It is about the integration of test automation in the fundamental DevOps activities and into the CI/CD pipeline, focused on getting feedback on the system’s quality before and during delivery and deployment.

In specific situations not all automated testing will be within the scope of the involved teams. For example, when testing COTS systems, SaaS systems, legacy systems, or with a particular pipeline of which certain test varieties (such as unit testing) cannot be a part. In that case teams can collaborate as a virtual team or separate teams such as system teams or shared services can take that responsibility. 

Continuous testing puts the proper amount of test automation (based on the risks to be covered) in the CI/CD pipeline. To achieve continuous testing, it is essential to focus on executing the proper tests at the right time and speed. A well-thought-through test strategy therefore is essential.

Auto-generation of test cases 

A step that will help immensely in moving towards continuous testing is the auto-generation of test cases. Today, the creation of test cases is a time-consuming, manual process and auto-generation of test cases can cut down this time significantly while adding value to the testing process. Moreover, auto-generation techniques can focus on designing test cases based on user requirements, rather than the created code. This approach has proven to be helpful in improving product quality and reducing rework as the tests are designed with the intended functionality in mind. Many organizations have already moved to using techniques such as model-based testing with the help of tools commonly available on the market.  

Cloud adoption and virtualization of test environments 

Another step that will help move towards continuous testing is cloud adoption by virtualizing the entire test environment. Additionally, DevOps teams do not need to provision entire environments as they can use service virtualization tools to virtualize parts of the environment that are either unavailable for testing, are still in development, or are third-party services. With automation, complete, ephemeral (volatile) test environments can be created on demand and decommissioned when testing is complete, reducing both the time as well as the expenses associated with maintaining environments. Cloud technology can help with this, and the adoption of cloud for the creation of virtual test environments is a logical next step for automating the delivery pipeline. For more information about infrastructure please refer to the chapter on DevOps Infrastructure.  

How to achieve continuous testing? 

To achieve continuous testing, there are certain prerequisites that must come into place:  

• Automation needs to happen within all selected test varieties, but keep in mind that 100% test automation is not a goal in itself.  
• An infrastructure that supports the ability to manage test data. 
• Testing has a business value – and a quality risk-based approach. 
• The ability to create environments quickly and remove them if not needed anymore. 
• The ability to populate the environments with the right data. 
• The ability to auto-generate and maintain test cases. 
• The size of the test set is small enough to be run with a short duration. 
• Integration of test automation solutions. 
• All DevOps people need to be involved in quality engineering. 
• Chosen tools match the organizational as well as technical requirements to be used efficiently. 
  In practice, a “big bang” implementation of continuous testing apparently does not work. The following approach is recommended: 
• Start small (but with the end goal in mind)  
  Instead of applying continuous testing across all DevOps teams, organizations should look at identifying a single team that is willing and able to adopt the practice and serve as an example of success to the rest of the organization.  
• Get baselines  
  If you do not have something to base things on you won’t know how to improve. Once you have your baselines (using metrics), you will know where your biggest pains are, and you can resolve them.  
• Over time, use the collected data to make improvements. Add automation where it will have the biggest impact. Eliminate manual handoffs. Over time, you will see how much you have improved, and which areas need additional attention.

Test automation solutions 

There are a number of ways test automation is currently done. Depending on the time already spent by an organization this ranges from “green field” situations, where test automation has been discarded in the past or never got started due to technical challenges, to a variety of legacy or unstructured test automation approaches.  
Current test automation approaches can be grouped as follows: 

• Framework automation  
  A test automation solution built in a single test execution tool. (e.g. Micro Focus UFT, Selenium)  
• Hybrid framework automation  
  Managing the test automation solution based on a separate test management tool. (e.g. Micro Focus or Broadcom tool combinations)  
• Tribrid framework automation  
  A combination and extension of the previous approaches results in a test automation solution providing the ability to be run stand-alone (single commandline test case execution), from a separate test management tool (as part of formal acceptance testing) as well as triggered as single test cases or scenarios from build and deploy tooling as part of the DevOps pipeline. The test cases become stand-alone code artifacts that can be executed as desired, as part of a larger whole (e.g. Robot Framework. Cucumber, Fitnesse)  
• Scriptless framework automation  
  Building test cases out of predefined building blocks available to users with all levels of expertise. (e.g. Tricentis Tosca, Sikuli)  

Test orchestration

The need for continuous improvement of test speed, coverage and quality has led to a situation in which organizations have “islands of automation” in their IT delivery lifecycle that are chained together with manual processes. Since the entire system can move only as fast as its slowest component, the whole process is unable to scale up to the demands of efficiency and speed.This situation is further complicated by increasingly complex architectures and the lack of visibility for all teams into the development pipeline.  

The quality and test policies are enablers for test orchestration as they align test automation activities over teams, which is a precondition for gathering information for real-time end-to-end-focused dashboards that inform about the confidence to establish the pursued business value.  

Eliminate “islands of automation” 

The next step is to address silos of automation with orchestration. In the realm of testing, test orchestration eliminates “islands of automation” by combining manual and automated tasks in a holistic fashion. It links together individual, automated tasks, which helps organizations move from spending time on manual handoffs, dependencies, wait times, and cycle waste to one in which test generation and execution are fully integrated as part of a fully automated and optimized continuous delivery (CD) pipeline.  

“Quality at speed”  

Testing today is more a sum of moving parts that need to be orchestrated together perfectly to achieve “quality at speed.” Organizations also want to orchestrate tests as part of a release. Test results and quality data can serve as feedback for the quality and risk of the release, as well as the efficiency of IT delivery as a whole. Orchestration also means that the teams must think about the best and fastest way to get feedback. This means on what level we automate the testing and what tests are run at what time, not all tests need to run every time. 

In this context, one way to better view the continuous testing within the CI/CD pipeline is to use customized dashboards, a practice adopted by many DevOps teams and organizations. These dashboards give an insight into whether the quality risks that need to be covered have indeed been covered before (parts of) the IT system is (are) deployed to the following environment.  

Another promising development is the rise of artificial intelligence (AI) technologies that provide “smart” test orchestration. In the near future, with the addition of machine learning capabilities, systems will be able to automatically determine the tests required in the release and production cycles.    

How do we realize test orchestration? 

In order to realize test orchestration in DevOps: 

• Create visibility of quality processes by implementing customized QA dashboards across the CI/CD pipelines. 
• Ensure that DevOps teams are properly supported by specialized people for tasks that the teams don’t have the knowledge and/or capacity for or which are at another organizational level (an example are end-to-end-regression-tests-on-demand by a service team).  
• Optimize test automation tools and test operations across DevOps. 
• Set up automated testing cycles to shift left and identify faults early in the lifecycle, thereby enhancing product quality. 
• Automate the self-provisioning of test data. 
• Automate provisioning of test environments with virtual services and test data. 
• Leverage artificial intelligence and machine-learning technologies to optimize test cycles. This may include selecting the correct tests to be run according to what is in the pipeline, to provisioning test environments and running the tests. Using natural language programming, the test orchestrator can learn the pattern of test data usage and auto-generate test data with minimum manual intervention. 

Good orchestration helps create a coherent QA and test strategy across teams and optimizes the use of test automation and test environments. It also gives business owners a continuous view of the actual E2E state of quality in their core processes and applications. This is an approach that covers each activity and task in the release. These activities incorporate both dev and testing and allow testing tasks to be incorporated into each activity in DevOps. Audit trails are compiled as a result of these fully tracked processes. As the release progresses through the orchestrated pipeline, real-time information is available to every person in DevOps in a single source of truth. As each task is completed, it can be automatically moved to the next task if all requirements are met (i.e., if all tests are passed) or the person or team responsible for the next manual task can be automatically alerted, and reminders and alerts are provided and escalated if tasks are taking too long. This shortens wait times and encourages collaboration across teams. 

The future: smart automated frameworks 

Test automation has been around for about three decades. Most automation frameworks are designed to automate manual steps but are not intelligent. They are unable to react to changes, dynamically generate the resources they need, or understand and interpret results. 

In our view, the next step in test automation will be if people think of it less as a capability, and more as a platform – as a broad, connected, and intelligent space. When all tools and functions occupy a common environment, they can talk to one another to drive a full lifecycle value proposition, rather than remain focused on just one particular activity. When that point is reached, test automation will be smart, and broader, business-driven benefits can be given the priority they deserve.  

When you want to design a smart automated framework (refer to The World Quality Report), keep in mind that an intelligent automation framework:  

• Is intuitive  
  It can check the code and create appropriate automated tests corresponding to the code changes.  
• Is dynamic  
  For example, it can use cognitive computing techniques to identify and screen elements dynamically, and update object repositories.  
• Can generate its own environment 
  For example, a framework can spin up environments at runtime through machine-readable definition files. 
• Can prioritize 
  A good use case for this is a framework that can identify and execute critical test cases from an automated suite to achieve high fault yield per test case execution using algorithms such as random forest algorithm. 
• Provisions its own test data  
  The data can be provisioned either by virtualizing, subsetting, or creating synthetic data.  
• Supports continuous improvement of the test set and the test process. 

“Everything as code” automation 

There is of course more than test automation in DevOps. Developing software efficiently, especially with cloud, requires one to automate everything that is repeated. This is done by defining the following as much as possible as code:  

• Infrastructure as code  
  Every piece of infrastructure, such as virtual machines, web apps, container registries, key vaults and so on must be defined with ARM templates (Azure), CloudFormation templates (AWS) or similar. This will enable to effectively version control the entire infrastructure. When a change of the infrastructure is required, the source template must be changed, not the target template, which in turn will be deployed using continuous deployment.  
• Configuration as code 
  As infrastructure is deployed, there is also need for software configurations and installations on top of the infrastructure, such as ZooKeeper, Consul, DNS, or SSL certificates. By configuring these, the entire pipeline is automated, and no manual steps are required. This ensures that the desired state of services are always defined, met, and source controlled.  
• Pipeline as code  
  By also defining the entire pipeline, by pointing out infrastructure, configurations, and ultimately the software code and how it is deployed, checked into source control as everything else, the team spends a maximum amount of time developing functionality instead of managing deployments. The pipeline configuration also defines test procedures, quality gateways and so on (test as code).  
• Documentation as code  
  All documentation should be versioned in source control using a common markup language, such as Markdown and Doxygen. This enables the versioning of the software to follow the documentation versioning. Markdown or Doxygen can then be rendered to HTML or PDF, depending on the users’ needs.