Click here to view the complete list of archived articles

This article was originally published in the Fall 2008 issue of Methods & Tools

Outsourced Testing – Friend or Foe?

Silvio Moser, http://www.swissq.it

It is common wisdom that if an activity is not a core competency of an individual, team or company, you either build up your skill set or have an expert third party perform the work for you.

Let me illustrate this. Doing the laundry is an activity I master. Separate the dirty clothes by color and sort out the delicate. Don’t forget to empty the pockets, if you don’t want to launder money. The actual act of washing is simple. Load the pile of clothes into the washing machine, choose your cycle, and start. When it comes to ironing though, I am clueless. I will spend half an hour pressing a shirt and it will still be full of wrinkles. No wonder I hate it. I therefore started buying easy-iron or non-iron shirts. While the latter don’t quite live up to their name, it did make the job somewhat easier. But even then some creases remained and I still have a number of "legacy" garments in my closet. The logical conclusion was therefore to outsource this activity to the dry cleaners.

Similarly, testing is an essential part of software development, but it is not the core competency of most companies. As of 2008, the (U.S.) average for defect removal efficiency is about 85%, meaning that 15% of total defects are found after software applications are delivered to customers. Best in class organizations have defect removal efficiencies that top 95 percent [1].

"Wrinkled" software though is becoming less and less acceptable in an ever more demanding and competitive market, considering that it costs at least ten times more to fix a bug once the product is shipped than to fix it in the development process, damages to the company image aside.

So, why not outsource software testing?

Outsourced Testing is a surprisingly wide-ranging subject area, as there are a multitude of testing tasks that can be carried out by a third party, from test strategy development to penetration testing.

While acquiring testing services from outside contractors isn’t unusual, its importance has increased considerably, as have the importance and maturity of testing itself. This is shown by the fact that the test outsourcing market is no longer driven solely by independent testing companies. All the major outsourcing companies now offer testing services as well.

Through outsourced testing, the following main benefits can be achieved:

• Speed up testing: test execution, located at the end of the software development life cycle can be a bottleneck, especially if development is already late. To make the most of the time available, a thorough test preparation and readily available testing staff are needed. These resources can by procured by an outsourcing provider. Also, in some cases testing can be sped up by taking advantage of the time zone differences, testing at night what has been developed or debugged during the day.
• Improve quality: The need to adhere to a formal testing process and to provide a proper test basis in the form of well-documented and stable requirements will result in finding more bugs, therefore improving the quality of the end product. In addition, some of the bugs will be found earlier, when analyzing the requirements or reviewing the test cases.
• Reduce testing costs: the main driver for outsourcing is often cost reduction, by moving operations to low-cost countries. Also in testing, costly IT and business resources can be freed from repetitive and time-consuming test activities as, for example, regression testing. The improved software quality can also lead to cost savings.

Let’s now discuss some of the particularities of Outsourced Software Testing. Remember that most, if not all, of the principles that apply to outsourcing in general also apply here and must also be considered.

Outsourcing resource- and time-consuming test activities, not only allows for greater capacity, scalability and flexibility, but also has the additional benefit of an independent validation. By that I mean testing that is not carried out by the development team itself (typically unit & integration testing), but by a separate test team.

But greater and/or cheaper capacity isn’t the only reason for outsourced testing and a strong case can be made for outsourcing testing tasks that require specialist know-how, for which you don’t have the skills or only need on a one-off basis.

When deciding on what parts of testing to outsource, you will have to look at it from different angles, mainly at the dimensions of test levels, test types and test activities:

• Test Levels: low-level testing (Unit & Integration test) is generally carried out by the developers themselves. If development is outsourced, these activities are outsourced with it. System testing on the other hand can be performed by an independent test team and is therefore an excellent candidate for test outsourcing. Finally, Acceptance testing requires business know how (for user acceptance) and a production-like test environment. It is therefore difficult to outsource.
• Test Types: generally speaking, all types of testing, both functional and non-functional, can be outsourced. By its nature, regression test is a good candidate for cost saving, because it involves regular repetition and test automation. Know-how intense test types like load & performance, usability and security are best outsourced to a specialist organisation, on a case-by-case basis.
• Test Activities: defining what activities within the test process should be outsourced (e.g., test planning, specification, execution and reporting) requires a strategic decision on how much control and knowledge is given away. It can range from test execution only, to performance of the whole process.

A typical example of such an analysis is shown in the following graph:

Not only do you have to decide on what parts of testing to outsource, but also which projects are most suited for it. If it is the first release of a totally new application, it may not be ideal to outsource its testing. However, if the project is an enhancement version of a stable product, with existing test cases, it would be a very good candidate. Possible selection criteria include:

• technical complexity
• in-house know-how
• maturity of requirements and test cases
• availability of test data

Outsourcing is never a simple task and many such efforts will fail because they haven’t been managed thoroughly. Unrealistic expectations, the selection of the wrong partner and communication problems are common reasons for failure. It is therefore imperative to use a well-thought out and planned procedure, that makes use of best practices and actively addresses the risks involved.

In the first phase you have to look at the feasibility of your outsourcing effort.

Beside the decision on what to outsource (as discussed previously), this includes the assessment of the maturity of your test process. Outsourcing is seldom a solution for internal process problems. In the contrary, the problem is outsourced and stays unresolved or even becomes accentuated. Identify and address these deficiencies first. Outsourcing may still be possible but might involve additional costs.

This leads us to the calculation of a rough business case. Consider different outsourcing options, from bringing in external testing experts to setting up an offshore development center, to find the most cost-effective alternative.

If you decide that it is all worthwhile, you can start the search for a suitable outsourcing partner by developing a request for proposal and inviting offers. It is important to look not only for testing know how, but also for product and technology knowledge. This will result in a steeper learning curve, allowing you to become productive sooner.

The selection of the provider will then kick off the transition phase. It is all about paper work and getting to know each other. While the lawyers do the contractual work, start defining the governance and split of work, as well as setting up the infrastructure. Start with a small number of projects, to gain experience.

All going well, the transition phase will lead you to steady state operation, typically within 6 to 9 months.

A key ingredient for success in day-to-day operations is an effective service level management. Identify the key performance indicators, define the service levels to be achieved and describe how these will be measured (what, who, when/how often) in a service level agreement. Besides the usual metrics such as on-time delivery, estimation accuracy and personnel retention, test specific metrics include:

• relative test effort
• test coverage
• percentage of test automation
• remaining defects

Lessons Learned

Don’t underestimate the time and effort necessary to set-up and transition. It will probably take you longer then you expected. Adapt your business case accordingly and ensure you have strong management commitment.

It will also take some time before you can fully harvest the benefits of your outsourcing effort, so keep looking for improvement potential and conduct periodic reviews with the provider’s management team to review progress as defined in the service level agreement.

The setup of the test infrastructure is another cost and success factor for outsourced testing. Setting up a full-blown test environment at the provider’s site is an ideal solution, but it is often difficult to replicate your environment, especially with regard to interfaces to peripheral systems and third-party products. In any case, a reliable and secure communication link is needed. This raises data privacy issues about giving access to your test environment and sharing test data. Only grant access on a need to know basis and use anonymous, or even better, synthetic test data, if you want to avoid surprises.

Remember though, that this is not primarily about technology, it is all about people. Communication problems, aggravated by geographical distance, cultural differences and different languages, are a major reason of failure. Engaging an onsite coordinator will facilitate communication with the offshore team. Well-defined processes for the creation and handover of test deliverables as well as defect-reporting will help minimize the potential for friction in daily work.

To help avoid misunderstandings it is important to define quality gates at the interfaces between you, the customer, and the provider - in both directions. Quality gates define what has to be delivered and in what quality. They usually involve one or more reviews. A detailed and understandable requirements specification is the key to each test (outsourcing) project. Make sure that the test team has understood what it is testing against. Later on, review of the test plan and test cases will help to establish if the testing effort is on track.

What is the quintessence? Outsourced testing can speed up testing, improve quality and reduce costs. But, no big surprise, it won’t fall into your lap – it requires a well-managed effort before, during and after the transition.

References

[1] Measuring Defect Potentials And Defect Removal Efficiency,
Capers Jones, Software Productivity Research, LLC http://www.stsc.hill.af.mil/crosstalk/2008/06/0806Jones.html

[2] ISTQB Standard Glossary of Terms used in Software Testing
http://www.istqb.org/

Back to the archive list