Control | DevOps

Definition

Control is to take adaptive measures, based on monitoring information, to ensure proper behavior of the IT system throughout the IT delivery cycle.

The monitoring information is used to prevent failures by forecasting the evolution of the quality level. If the quality tends to go below a specified threshold, corrective measures can be taken.

Monitoring is about continuously gathering feedback, by measuring indicators, about the IT System, the IT delivery process, and the team (in other words: products, processes and people). Based on this feedback, the team forecasts the (possible) behavior of the system, process and team. When this forecast shows a potential problem in the (near) future, because the indicators would go beyond the tolerances, it is necessary to take adaptive measures. The process of taking these adaptive measures is what we call control. With control, the team ensures the expected behavior of the IT-system throughout the IT delivery cycle. They also ensure the right quality of the IT delivery process and the team. In control, we distinguish two different areas:

Technical control measures
Organizational control measures

The following sections describe these two areas in more detail. Please be aware to start control actions as early as possible, don’t wait until there is no more possibility to change in the right direction!

Technical control measures

Technical control measures are quality measures that use software and hardware to prevent and solve (potential) problems and ensure that the DevOps team is in control of the delivered IT-System and the IT delivery process.

Examples

Examples of technical control measures:

Load balancing:

When there are lots of requests for a specific server, there is a risk that this server is flooded with these requests. To reduce the risk of flooding, a load balancer can be put in place to spread all requests over two (or more) servers.

Graceful degradation:

When monitoring shows that the reliability of certain parts of the IT system is below a certain threshold (for example because of unexpected high volumes of transactions), then the automated control system will turn off certain, less important, features to attempt to keep the core functionality of the system available to the users.

Code coverage to ensure maintainability level:

When you use code coverage as an indicator, code coverage may drop below a certain threshold when this system is changed or extended. This would decrease the maintainability of the system. Adding more automated (unit) tests is a technical adaptive measure to raise the code coverage above this threshold (see section 46.8 for more info about code coverage).

Ideally, the above mentioned (and many other) technical measures are included in the CI/CD pipeline so that both monitoring and control are automated to a large extent.

Organizational control measures

Next to technical control measures, there are organizational control measures. Where technical control measures are executed by software and/or hardware, organizational control measures are put in place by various roles in the organization.

Who will initiate control activities? Since in a high-performance cross-functional team there is no distinct leader or manager (like we would have in sequential IT delivery), specific roles are responsible for initiating control.

This can be illustrated using the CI/CD pipeline.

Control activities related to the overall process (see “orchestration” in the above figure) are initiated by the end-to-end quality orchestrator. This role is defined as: “The person responsible for organizing end-to-end quality.”

Initiating control activities related to the team scope (the CI part of the build pipeline) is the team’s responsibility as a whole. For example, based on alerts from testing or retrospective outcomes, the team members initiate control actions.

Initiating control activities related to the release pipeline (the CD part) is the orchestrator’s responsibility in collaboration with representatives of all teams involved. For example, based on alerts from end-to-end testing.

Examples

Examples of organizational control measures are:

Change the product
If the product as it was required can not be delivered in the timeframe then the team can change the backlog and take low priority user stories out of their backlog and postpone it to a later IT delivery iteration.

Change the process
If the team cannot perform certain tasks themselves they can get support from people outside the team, such as from a system team, shared services or other kinds of supporting people.

Change the people
If the team members lack certain skills they can arrange coaching or training, or new (skilled) people can be added to the team.

Let us, as an example, look at the following situation: At some point in your IT delivery process, some human interaction with the outside world occurs. Only one person can do this interaction. When there is more work to be done than this person can handle, congestion will occur. When monitoring is in place, the DevOps team is notified before this happens. The team(s) involved can add a second person to handle the tasks involving human interaction as an organizational adaptive measure.