Click here to view the complete list of archived articles

This article was originally published in the Fall 2004 issue of Methods & Tools

---

Managing Your Way through the Integration and Test Black Hole

Ellen George, PS&J Software Six Sigma, www.SoftwareSixSigma.com

The Black Hole

How do you know how much effort to schedulefor integration and test? Once in test, how do you know how much longer it willtake? How do you know when you have found all the defects that you can find?When is integration and test really done? Often, integration and test is seen asa black hole. The product enters the phase, but no one seems to know when it will come out.

In order to understand how to accurately estimate integration and test effort and to develop a meaningful defect removalstrategy, we need to first examine the relationships within the development and test cycle relative to defect injection and removal.

Integration and Test Processes

For simplicity, the discussion in this paper is limited to software only. However, the concepts can easily be extended to thesystems engineering life cycle phases as well. In software, the major softwaredevelopment life cycle phases are requirements, architecture, design and code,ref. Figure 1, The Development and Test Cycle. In addition to creating product,these life cycle phases represent the primary mechanism for injecting defectsinto the product. Conversely, the major product validation life cycle phases arecompilation, unit test, integration, and system test. The purpose of these lifecycle phases is not only to validate the product, but also to remove the defects injected during development.

Figure 1 - Development and Test Phases

Each of these defect removal phases can be paired with a corresponding product development phase. The defect removal phasesshould be designed to find the unique category of defect injected during thecorresponding development phase. Compile is used to find the syntactical defectsinjected during the coding phase. Unit Test is used to find the logic errorsinjected during Design. Integration should be used to find and fix the interfacedefects injected during Architecture, and finally, the purpose of System Test is to find Requirements defects.

Ultimately, the goals of integration and test are to:

• bring together the multiple pieces of a system
• find and fix defects that couldn’t be found earlier
• ensure that the requirements of the system are met
• deliver a product of predictable quality that meets the business’s quality objectives as well as the customer’s quality expectations.

Estimating Integration and Test Effort

Many organizations estimate integration effort based solely on the number of test cases to be executed (integration effort =number of test cases x average time to run a test case). The problem with thismethod is that it misses the main cause of integration and test overruns …defects!! A better technique for estimating integration effort is to base it onthe expected number of defects to be found in the product during integration andtest plus the time it takes to run through a clean execution of all the testcases. Thus, integration effort = (number of predicted defects x average time tofind and fix a defect) + (number of test cases x average time to run a test case).

In order to use this technique, you need to a mechanism to estimate the expected number of defects in the product and todetermine the average cost of finding and fixing a defect during integration and during system test.

There are only three types of measurements required to manage the integration and test processes: effort, size and defects."Effort" is the effort expended individually by each of the testingparticipants to find and fix the defects found in test. "Size" is thesize of the work product being tested, often measured in lines of code (LOC)."Defects" is the quantity, type, injection phase, removal phase, anddescription of the defect. Although a defect measurement consists of severalpieces of data, it is collected as a single measurement at a discrete point in time.

Using these three measures, we have all the data we need to estimate the number of defects expected to be found inintegration and in test as well as the average cost to find and fix them.

Predicting Product Quality

Let’s assume that the data collected tells us that developers inject approximately 40 defects per thousand lines of code(KLOC) in design and 60 defects per KLOC in code. Let’s also assume that eachdefect removal step typically removes between 35% and 50% of the defects thatare in the product. Using these measures, we can begin to model the productquality as shown in Table 1, Quality Matrix. Note the values in Table 1 are allreferenced to 1 KLOC of new and changed code. These numbers are used for illustration only. Your own data may differ significantly.

Phase	Defects Remaining	Defects Injected	Defects Present	Defect Removal Yield	Defects Removed	Cost/ Defect
Design	0.0	40	40.0	0%	0.0	5 min
Design Inspection	40.0	0	40.0	50%	20.0	10 min
Code	20.0	60	80.0	0%	0.0	1 min
Compile	80.0	0	80.0	50%	30.0	1 min
Code Inspection	50.0	0	50.0	50%	25.0	5 min
Unit Test	25.0	0	25.0	50%	12.5	12 min
Integration Test	12.5	0	12.5	35%	4.4	5 hrs
System Test	8.1	0	8.13	35%	2.8	10 hrs
Customer	5.3

Table 1 Quality Matrix

To determine the average time to find and fix a defect in each of the defect removal phases, take the total effort spent inthe defect removal step and divide by the total number of defects found in thatphase. Assuming all defects that were found were also fixed, then the averagetime to find and fix a defect in integration would be the sum of the time spentby all testers in integration divided by the total number of defects found andfixed in integration. The same can be done for system test. This data can bedocumented in the quality matrix in the column titled "cost/defect".Note, the time spent writing the test cases and the time spent doing the finalclean run through the test cases should be subtracted from the total integrationor test effort before doing the calculation.

Clearly, the number of defects remaining in the product when it reaches integration will depend on the quality of thedeveloped product as well as the adequacy of the defect removal steps leading upto integration. It is critically important for the developers and the testers toagree upon and set goals for defect injection and removal phases at the start ofthe development effort. These goals should be documented in a Quality Matrix.The effort estimate for integration and test can then be calculated bymultiplying the number of defects expected to be found in integration and testby the average cost to find and fix a defect in those phases.

Quantitative Entry and Exit Criteria

What happens, however, if the quality of the product when delivered to integration does not meet the plan as documented inthe Quality Matrix? If more defects enter integration than planned, then theestimated effort will not be adequate to complete integration and test.

Entry and exit criteria are used to "gate" the process. Before the product is allowed to exit a phase, thenumber of defects removed should be compared to the goals as documented in theQuality Matrix. A threshold can be set around the expected defect removal value.If the number of defects found in-phase falls within the expected range, thenthe product is allowed to move forward. If the number of defects found fallsoutside the expected range, then the product is prevented from moving forward.

For example, using the plan in Table 1, we expect to find 20 defects in design inspection. We can place a +/- 20% thresholdaround this value, resulting in an expectation of 16 to 24 defects per KLOCfound in design inspection. If the actual number of unique defects found fallswithin these bounds, then the product moves forward to the coding phase. If thenumber of defects found in design inspection exceeds the upper bound, then theinspection moderator stops the product from moving forward and takes steps todetermine if the inspection team did an exceptionally good inspection, or if theproduct was buggier than expected. Similarly, if the number of defects found indesign inspection is less than expected, then the inspection moderator needs todetermine if the team did a poor inspection, or if the product was of particularly high quality entering inspection.

One relatively quick mechanism for determining product and process quality can be made by asking another developer to spend afew minutes reviewing the fixed product. If they can find a defect in a shortperiod of time, then the product is probably buggy. If the product is determinedto be buggy, then corrective action needs to be taken before the product exitsthe phase. If a product that is known to be buggy is moved forward though theprocess, then each subsequent phase will have to spend more effort thanoriginally planned and budgeted to remove the extra defects. When a productmeets all of its phase exit criteria as it moves through the developmentprocess, then there is a high probability that it will enter integration with the planned number of defects waiting to be found.

Managing the Integration and Test Effort

Integration and test is managed by managing defects. The entry criteria into integration should be used to prevent poorquality product from moving forward. Only product that meets the entry criteriaas specified in the Quality Matrix should be accepted. If product that is knownto not meet the entry criteria must be accepted, then additional cost andschedule needs to be negotiated to adequately test the product. Once the producthas been accepted and integration has begun, several techniques can be used tomanage the defects that are found. A brief discussion will be provided on twosuch techniques, "Looking for Systemic Defects" and "Test vs. Rewrite"

Systemic Defects

As humans, we tend to be very repetitive in our actions. As a result, when we make a mistake once, it is probable that wehave made the same mistake elsewhere. This human characteristic can be used toan advantage. Each time a defect that is found in integration and test, itshould be used as an opportunity to look for additional occurrences of the same defect elsewhere in the product.

Each time a defect is found, a root cause analysis should be performed on the defect. When was the defect injected? Howwas it allowed to leak into integration? Where else in the code might a similardefect have been injected? By answering these questions and looking foradditional occurrences of the defect, the tester can maximize the number ofdefects found through a single test failure. An automated search or targetedinspection can easily be used to search suspected code for duplicate defects.

Test vs. Rewrite

Defects tend to cluster. Actual data that we collected over 1200 modules illustrates this point in Figure 2, Defects PerModule. 70% of the defects were found in 20% of the modules. 50% of the defectswere found in 10% of the modules. This data suggests that if we can identify the20% of the modules that are the buggiest and take corrective action, we might be able to manage, or eliminate, 70% of the defects in the product.

Figure 2 Defects per Module

When a particularly buggy product is identified, appropriate corrective action could be to remove the module from theintegration build and re-inspect it for additional defects. If too many defectscontinue to be found, then it might be more cost beneficial to rewrite it than to continue testing it.

Example

Assume Project A uses the quality plan specified in Table 1. Module X is currently in integration. Module X’s size is1 KLOC. According to the quality plan, a total of 4 to 5 defects are expected tobe found in integration for a module of this size. 10% of the module’sintegration test cases have been run, and 3 defects have been found so far. Wecan assume that since 10% of the test cases have been run, that the 3 defectsfound represent 10% of the defects that will be found in integration. So, wecalculate that Module X probably has a total of 30 defects that will be found inintegration, compared to the plan of 4 to 5. As a tester, what should you doabout this module? Continue to test it? Or send it back to development to be re-written?

Continue Testing

At 5 hours per defect, this module has already cost approximately 15 hours in integration. If we continue testing to find theremaining 27 defects, at 5 hours apiece, they will likely cost a total of 135 hours.

If we continue testing, then we can assume that:

• 30 defects will be found in integration
• we expected to find 35% of the total remaining defects in integration, so
• 86 defects entered integration (30 / 35% = 86)
• 56 defects will leak into system test (86 – 30 = 56)
• we expect to find 35% of the total remaining defects in system test, so
• 20 defects will be found in system test (56 x 35% = 20), and
• 35 defects will leak out to the customer (56 – 20 = 36).

The estimated cost of continued testing is 335 hours:

• 27 remaining defects x 5 hrs = 135 hours remaining in integration
• 20 defects x 10 hours = 200 hours remaining in system test
• 36 defects delivered to the production

Re-Write the Module

Assume a developer can design, code and unit test code at a rate of 20 LOC per hour. If the quality of the product meets theentry criteria for integration, then there will be approximately 5 defects found in integration and 3 defects found in system test.

The estimated cost of re-writing the module is 105 hours:

• 1000 LOC x 20 LOC/hour = 50 hours to re-write the module
• 5 defects x 5 hours = 25 hours to integrate the module
• 3 defects x 10 hours = 30 hours to system test it
• 5 defects delivered to the production

In this example, throwing away the module and re-writing it would yield a net saving of 230 hours over continuing to test it.

Summary

In order to successfully manage integration and test, it must be planned and managed as an integral part of the entiredevelopment lifecycle. The factor that most affects the effort required tocomplete integration and test is the number of defects expected to be found inthese phases. This number comes as a direct result of the development process used to generate the code to be tested.

The key steps to a successful integration and test effort are:

• Collect basic measurement data: effort, size and defects. Use the data to create a Quality Matrix to baseline the organization’scapability.
• Build a project specific quality plan based on the organizational Quality Matrix. Get agreement on quantitative phase entry andexit criteria from both developers and testers during project planning.
• Estimate time required for integration and test based on (number of expected defects) x (average time to find and fix a defect)
• Ensure that both developers and testers agree to deviations from the quality plan. Deviations from the plan should also result in negotiated adjustments to the budget and schedule.
• During integration and test execution, whenever a defect is found, use it as an opportunity to find as many additional defects as possible. Since people tend to be repetitive in their actions, it is very likely that the same defect will appear within the code in several more locations.
• Use defect identified in integration and test to understand how the defect was injected and how it was able to leak into the later life cycle phases before being found. Take corrective action to either prevent the category of defect from being injected again in the future or to contain it in the phase in which it was injected.
• When defect data on module indicates that it may be much buggier than planned, return the module to development for corrective action. Use calculations based on productivity rate, expected defects and time to find and fix a defect to determine the best course of action. Consider either a re-inspection of the code or to possibly re-writing the code from scratch.

Back to the archive list