Two Faces of End-to-End Testing: Horizontal and Vertical

End-to-End (E2E) testing is used as a term to refer to certain test approaches. Misconceptions about its scope lead to confusion. Many organizations refer to E2E testing without specifying whether they mean horizontal or vertical E2E, which are distinct approaches with different focus, planning and scope. This article aims to clarify these concepts and highlight the importance of defining the scope of E2E testing.

End-to-end test: A test variety where the end-to-end (E2E) functionality of a business process, supported by one or more systems, is tested.

It’s critical to visualize the scope of the E2E process when working with multiple systems. Before you begin, make sure to understand, for each process, what the process is supposed to achieve, where it starts, and where it ends.

Vertical E2E Testing

Vertical E2E testing delves deep into the individual architectural layers of a system. This approach ensures that each layer, from the frontend (GUI) to the backend, operates correctly and integrates seamlessly with other layers. Vertical E2E testing is particularly useful for identifying issues within specific components or services that may not be apparent in horizontal testing. Note that not all vertical E2E processes include a frontend layer.

Vertical end-to-end test: focus on E2E business processes per system. (common synonyms: Application/ Domain/ Module/ Solution train/ Stream/ Line of Business (LoB)/ 3rd Party/ Legacy)

Key aspects of vertical E2E testing include:

System-layer-specific scope: Testing each layer of the individual system, such as the presentation layer, business logic layer, and data layer.
Component integration: Ensuring that individual components within each layer interact correctly.
Technical depth: Focusing on the technical aspects of the system, such as database operations, API integrations, and server-side processes.
Scope and coverage for IT delivery team: the responsibility of the coverage is for the IT team that develops and maintains this specific solution.
Avoid dependencies: Apply service virtualization and test doubles to avoid dependencies outside the scope of control, such as drivers and stubs.
Fast and continuous execution: With fewer dependencies and a more controlled setup, test execution becomes faster, more reliable, and available on-demand.

Horizontal E2E Testing

Horizontal E2E testing focuses on the user journey and overall business logic across different (legacy) systems, (3^rd party) interfaces, SaaS platforms (such as SAP, Salesforce or Dynamics CRM), and authorization. Horizontal E2E testing is a type of functional testing, mainly part of the User Acceptance Test phase. The approach simulates the full flow of information within the eco-system by executing end-to-end business scenarios. Preferably conducted by key stakeholders, such as end users responsible for their part of the process. Important other stakeholders, like IT delivery teams, platform teams, and virtual teams have to support horizontal E2E testing. The results provide information about data integrity and whether all integrated components work together as expected. For instance, in an e-commerce system, horizontal E2E testing would verify the process from product selection to payment and order confirmation, covering multiple systems such as the website, payment gateway, and inventory management.

Horizontal end-to-end test: focus on multiple systems that together support one business process.

Concrete example:

Key aspects of horizontal E2E testing include:

Cross-System integration: Verifying that data flows accurately between different systems and components.
Focus on key business needs: Covering key business processes real-world user scenarios to identify any potential issues that could affect the user experience.
Complex coordination due dependencies: test data management, test environment, test scenarios, test execution, software availability and testability. Moreover, software synchronization across multiple systems and teams is complex. Assigning a dedicated E2E coordinator or E2E orchestrator is a good practice.
Platform E2E team: it is a good practice to assign a group of people from the respective IT delivery teams, supplemented and supported by business people that focus on the coverage and execution of horizontal E2E tests.
Apply explicit entry criteria: It is not advisable to start with a horizontal E2E test when individual systems or their integrations and interactions are not yet tested. Or not yet on the required level of quality from user acceptance perspective. Align and approve these criteria upfront with all important stakeholders as the unavailability will impact the planning.
Expensive and complex planning and execution: Test Automation for the most critical horizontal E2E processes is recommended for regression testing, though complex due to needed business knowledge, resource skills and availability, continuous changes, test data, and test environments. Unfortunately, manual execution remains common practice.
Authorizations: Horizontal E2E is often one of the last steps in the functional testing journey. Make sure to validate the horizontal business process with the correct authorizations in place. Meaning, execute each step of the business process in each dependent system with the correct business roles. Logging in and out of the different systems and execute the business processes with the correct business role and authorization, is an essential part of this process.
Scope: The horizontal E2E scope should be determined and aligned with the main business representatives to discuss which process to test, with what data, and what to validate at the end.

System integration testing

Before transitioning from vertical to horizontal E2E testing, ensure all individual system integrations within the horizontal E2E business process are available and thoroughly tested. Skipping this step may lead to issues during horizontal E2E testing. It is advisable to conduct dedicated system integration tests for each interface.

Together with horizontal and vertical E2E testing, this provides information quality at three levels: the system, system integration, and E2E business process. A balanced testing strategy, designed to identify issues at the right time, results in quality at the right time.

Tips for E2E Testing

It’s important to state that horizontal E2E testing should have fewer tests compared to vertical E2E. A quality risk analyses will help to identify risks and determine the most critical E2E processes. Note however, that fewer tests don’t mean less coverage. Horizontal E2E testing should be done on a high-fidelity/production like environment. Fidelity is the indication of how much a test environment resembles the production environment.

Determining horizontal end-to-end test cases may seem very difficult, especially in situations where a complex mesh of systems is the test object. However, one way to approach this is to start defining the desired outcomes that are needed to build confidence in the end-to-end-process.

Then work backwards towards the start of the network of systems, determining what inputs are needed to get the desired outcome. This generally will result in more focused testing and a much lower number of test cases. Alignment with all stakeholders in the E2E process is needed to guarantee the needed test data is generated and available.

An alternative approach to identifying the most critical horizontal or vertical E2E processes is exploratory testing. This self-controlled approach leverages existing expertise to conduct value-based research, providing information about quality. The results of exploratory testing can help determine the necessary coverage to establish confidence in IT-products and their processes.

In summary, understanding the distinctions between horizontal and vertical E2E testing is essential for effective software development and pursuing confidence in the solution. By clearly defining the scope of E2E testing, preparing the test environment and determining test data to be used, development teams and business teams can achieve comprehensive coverage, ensuring the delivery of robust, user-friendly software that meets both technical and user requirements. This approach contributes to achieving the right quality at the right time.

Together with horizontal and vertical E2E testing, this provides information quality at three levels: the system, system integration, and E2E business process. A balanced testing strategy, designed to identify issues at the right time, results in quality at the right time.

Published: 31 January 2025
Authors: Boyd Kronenberg, Pepijn Paap