Author: Justin Uy, Sogeti USA
June 2010
The date is August 14, 2003 at 12:15 PM.Over the next four hours, a cascading sequence of failures will occur that results in a catastrophic loss of power for over 55,000,000 people across the American Northeast at an estimated economic loss of $7 to $10 billion. What could have caused such a disaster? A post-mortem investigation revealed that the root cause could be traced to a race condition in a third party software system. In the not too distant past, testing was relatively simple. Standalone information systems could be analyzed and tested via a discrete division of labor. Basic client/server architecture meant that most software defects could be traced to an issue on the client or database side. As the lines between applications blur, the associated information chains grow longer and more complex, and their importance and risk to the business have increased tremendously. A change at one level can have far-reaching consequences.Today’s typical computing environment has evolved into a mix of customized, legacy, COTS, and third party components. As these systems evolve in complexity, it is imperative that your approach to testing evolve with it.
Information Chains
To approach end-to-end testing, one must understand the concept of an information chain.Where a unit test might test the functionality of a single system feature, or a system test will test the full inputs and outputs of the system, an end-to-end test will demonstrate that a consecutive series of systems supports business process according to specifications. The path that information follows within systems and across system boundaries is called the information chain. Each individual chain within a series of systems carries its own risks and constraints on testing. An example information chain is visualized in Figure 1. Each small box represents a function within one of three systems. Information is passed between each function within or across system boundaries forming an information chain from function A to E.
End-to-End Testing Strategy
The first step to creating an end-to-end testing strategy is the development of an Integral End-to-End Test Plan (IETP).The goal of the IETP is to inform the interested parties of the chain risks and feasibility of the End-to-End test as well as to help those parties make an informed decision as to the execution of the test.To create the IETP, the following steps must be taken:
1. Inventorying processes and systems
2. Analyzing chain risks and determining feasibility
3.Determining scope, sub-chains, strategy and test environment
4. Planning and estimating
5. Creating the IETP
6.Go/No Go IETP
7. IETP Management
Inventorying Processes and Systems
This activity is to record the system stakeholders, as well as the relevant available data on the existing and to-be systems and processes.This process is greatly facilitated if either or both of the process/system landscape document or system interface overview is available.
If not, the necessary information can be obtained by interviewing all stakeholders for each individual information chain.
Analyzing Chain Risks and Determining Feasibility
Examining the individual risk of each chain and sub-chain is important to making a wellfounded decision concerning the execution of the end-to-end test as well as delimiting the scope and determining the optimum test strategy.
Determining Scope, Sub-chains, Strategy and Test Environment
This step in the end-to-end testing process depends heavily on your chain risk analysis and feasibility study.When determining the scope of an end-to-end test, it is important to carefully balance the cost and feasibility of testing a chain with the risk of not having enough coverage for your test.
When it is not feasible to run a full end-to-end test of a chain, it may be cost effective or convenient to divide it into various sub-chains for testing. This division may be done in any number of ways including the chain’s organizational aspects, business processes, geographical location of parts, technology clustering, internal versus external, chain length, planning aspects, chain risk, time pressure relating to lead time, specific responsibilities of individuals or the wishes of the project sponsor.
Following the definition of the scope and the sub-chains, it is important to determine an appropriate strategy for your application under test. In general, it will be most beneficial to allow for the sub-chains with the greatest risk to be tested first to allow the possibility of later retest.
Planning and Estimating
Now that the background research activity has been completed, it is time to look at the whole and determine the time and budget commitment that will be required to execute a test strategy.This activity is mainly comprised of three steps: determining the execution milestones, establishing and determining the lead time, and determining the start date of the end-to-end test.
Creating the IETP
After the planning and estimation step is complete, the creation of the IETP closely resembles the inception of many other types of test plans.These activities will include the determination and assignment of roles, creation of a communication plan, setting
up a consistent defect procedure and reporting.
Go/No Go IETP
After the IETP has been defined, the decision rests with the client/project sponsor as to whether the end-to-end test will be executed.The main sources of information for this decision making process are:
• The chain risk analysis
• Results of the feasibility study
• Overall strategy per sub-chain
• Overall planning and estimate
• Test process risks as collected
• Other information in the IETP
Despite seemingly being a yes or no decision, there are multiple possible outcomes for this step. For example, management may agree to the necessity and feasibility of the end-to-end test but disagree with an aspect of the proposed implementation by deciding on a hard shop production rollout date, or by deciding on a rollout day that will be determined by a zero-defect date.
IETP Management
After management has approved of the IETP, the test management duties become an ongoing process of three main activities: monitoring, reporting, and adjusting.The monitoring activity involves paying attention to the alignment of the project schedule with available resources to avoid possible conflicts. It also includes the analysis of new and ongoing project risks and the triage of defects.The reporting activity deals with the analysis of project risks and defects and suggests measures to mitigate both. The adjusting activity seeks to implement the mitigation strategies suggested by the reporting activity and is subject to management approval.
Conclusions
As technology continues to advance, the systems that drive today’s business processes grow increasingly more complex and interwoven. In a world where the state of a single variable can mean the difference between smooth running success and an absolute catastrophe, it is imperative that your test plan is not a piecemeal work, but a complete and thorough end-to-end test strategy.The techniques and procedures presented here as part of Sogeti’s TMap framework will lead you to the state where you can manage your testing efficiently as well as minimizing your overall risk.
For further reading on end-to-end testing methodology, refer to End-To-end testing with TMap NEXT by Rob Smit and Rob Baarda or contact your local Sogeti USA office.
Justin Uy is a consultant with the Managed Testing Systems Practice of Sogeti’s Dayton office.