Test execution | DevOps

The actual execution of a test can start at the point when the test object is deployed to the relevant environment. In a CI/CD pipeline the automated test will start automatically as part of the process. In other situations, a team member must start the execution.

Definition Test execution is the execution of tests by running the system under test and thus obtaining the actual results that can be compared with the expected results to determine whether the tests have passed or failed. This is part of dynamic testing.

The first step in test execution is to make sure the test object is properly installed and all preconditions (such as availability of the right test data) have been fulfilled. This is often done by performing a pre-test (also known as intake test or smoke test).  

Definition A pre-test is testing the delivered product in such a way that it is determined whether or not the product is of sufficient quality to execute a complete test of this product.

When the team has created test scenarios and/or test scripts the test execution will execute many test cases together and investigate the results after execution. When the team does exploratory testing, test cases are designed and executed iteratively, and the result is investigated after every test case execution (See figure below).  

Explicit and implicit tests

When executing tests, the tester observes the behavior of the IT system. With the explicitly designed test cases there are specified expected outcomes. However, the tester must also observe the behavior of the test object more generally. Specific aspects of the system are often tested implicitly, that is without creating specific test cases. Think for example of a tester that is testing an arithmetic function and while doing so also pays attention to the usability of that function without having explicit usability test cases.  

Definition An implicit test is observing behavior of a test object during execution of test cases that were not designed for testing that specific behavior.

Definition Explicit testing is defining and executing specific test situations and test cases for testing a specific situation.

Teams that automate tests in their CI/CD-pipeline should be very aware of the fact that an automated test only performs explicit testing. So, if there are aspects that would be tested implicitly in manual testing, explicit tests will have to be created to be able to have a valid assessment of the quality of the test object.  

View Building Block

Different test varieties have a different focus

A DevOps team will need a lot of variety in testing. Even when all testing is organized and performed together, still the team needs to be aware of this variety.  

Executing an API test for example, is very much different from a UX (user experience) test or an end-to-end customer-journey test or a non-functional test. Therefore, it is often wise to organize test varieties separately and to carefully consider which quality risks and business value are best tested in what test variety.  

Also, different test varieties will need different test execution schedules. Typically, tests are only executed if related parts of the system have been changed. If, for example, the user interface was changed but the API was not, then only UX testing need to be executed.  

What if the test cannot be executed?

There are many reasons why a test cannot be executed. The test environment may not work, another test may have failed and blocked all following tests, the test data may not be available, the test script may need to be adjusted because of interface changes, et cetera.  

The worst reaction to a test not being run is assuming that it would have passed. This may seem obvious, but teams that are working towards a deadline, for example the end of a sprint, may consider cutting some corners and still deliver. If a test cannot be executed, the exit criteria are not fulfilled and thus the team cannot deliver their changes. Therefore, the team will need to focus on proper maintenance of their test scenarios, test data and test scripts. Continuous testing not only refers to continuous test execution, but also to continuous maintenance of the testware.