Click here to view the complete list of archived articles
This article was originally published in the Summer 2001 issue of Methods & Tools
Automation Strategy Out of the Garage
Mark Rossmiller, SWQA Software Design Engineer,Hewlett Packard
Abstract:
As a software engineer, I have been involved in software quality and testing of various forms of commercially developed software for over seven years. Many of these software applications ranging from database tier servers, LAN|Web communication, HR applications, installers, and printer device drivers. In all of these software examples, I developed various forms and levels of automation to increase the efficiency of the testing effort with an equal variety of success. It has been my experience that most of the success implementing automated test tools has primarily not resided in the automation tools employed. Rather, it has existed more in the vision of the leadership to make a paradigm shift from old methods of testing software in order to actualize the value of automated test tools. Without an environment designed, accepted, and developed in all stages of the test planning process the success of automation can be plagued with delays, feature changes, and resistance to change. The following testing strategy was developed to assist my HP colleagues and partners to recognize the necessity to reduce the time and cost of testing methods as it currently exists. I have borrowed from the HP Rules of the Garage as a template in defining the paradigm shift needed to facilitating better, faster, and smarter use of software testing resources.
Believe you can change the world.
Past success can be an invigorating reminder of how good a job we have done; that there is no need to alter the course of how we test our products, or whether we need to investigate alternatives to the way we have performed testing on past products. In this view, past success can be a detriment to being able to change testing habits. To believe that one can change the world requires that we are willing to make change ourselves. I think Carly Fiorino stated this best, "We must be willing to eat our own dog food…". This means you cannot simply advocate the benefits of automation. One must actually implement and use the tools within a process improvement cycle – plan|do|check|act (the Shewhart cycle) (1 Grady). Automation development and testing must be understood as having no difference in application from that of the development practices involved in the software being tested.
Work quickly, keep the tools unlocked, and work whenever.
Although software testing is often constrained to follow the product development release schedule, it never-the-less should make every effort to optimize planning and arrange resources on its own well defined set of check points. Setup time must be reduced to a minimum or completed prior to drop of the next code release. It is essential to develop processes that enable testing as soon as release drops occur. This requires significant reduction in setup time associated with testing of the next software release cycle candidate. In most cases, there needs to be a consistent test environment established having static test machines in place at all times. In addition, a table of objectives and backup procedures should be prepared in advance to accommodate dynamic testing where a variety of SW|HW|OS configurations are necessary to complete the scope of planned testing. We cannot move quickly enough if we choose to wait until the next drop occurs.
Testing tools are only valuable if they get used. Particularly with complex automation, it is necessary for the tool users to become familiar with its use, provide for the expanded use of the tool, and most importantly, to provide usage feedback that further enhances the value of the tool in testing. It can then be determined if the support of the tool for required testing has merit or if a suitable alternative is available. Some of the risks in practice that can inherently devalue tool and automation investment are:
When testing resources become scare, further improvements in efficiency can still be realized through automation that can be run unattended or after-hours. "Lights out" testing has been developed and available for several years at HP. VCD has several automation tools built with Visual Test â on site and leveraged from other sites. These tools have great value, have a history of proven use, and should continue to evolve to meet changing product testing demand. (Please find tool detail in Resources section of this document).
Know when to work alone and when to work together.
"Lasting productivity improvement must come from within and cannot be imposed from without." (5 Bumbarger).
If you want to work alone in an organization, propose radical change. Obviously this is not the time to work alone since progressive change requires a receptive and collaborative effort. But the act of proposing change presents many perceptual, habitual, cultural, emotional, and political obstacles from those who would be expected participants and supporters. That is why these environmental factors need to be addressed or it is futile to introduce the tools of change. Real, lasting productivity improvement requires change. And change requires creativity and innovation. The irony in productivity improvement is that success is most unlikely to succeed when imposed from without – but often is forced to change when ineffective from within.
Share tools, ideas. Trust your colleagues.
Results of using automation in software driver testing showed a reduction in testing time by half. There is also the inherent benefit of consistency of execution – repeatability. Thousands of keystrokes or settings can be repeated exactly without the variation when done by hand. Automation reduces thrashing through defects that are not properly characterized and/or hard to duplicate. This results in thousands of reported defects over time remaining open and without a clear resolution. Automation further benefits the tester by providing time savings when multiple SW|FW|HW|OS configurations are required. A multiple of tests can therefore be run concurrently and multiply the effectiveness of the testers time.
No politics. No bureaucracy.
Organizations by the very nature of their existence are political and as they grow - become bureaucratic. Our organization is no exception. But this does not mean that visionary thinking cannot be used to improve or eliminate as many obstacles within the organization to effect positive change.
The customer defines a job well done.
From the standpoint of software quality, the customer as well as the next customer represents Argus project data, R&D partners, beta testing, outsource testing, and final media to name a few. Automation has indirectly provided improvements in servicing the next customer via automated defect handling tools to report defects more efficiently and earlier in the development cycle. Automated defect submittals have, for the past 2 years, contributed the largest share of reported defects at a savings of more than 771 hours in submittal costs. We should continue to support these systems and provide the technical resources and tools to maintain the reduced cost of defect handling.
Radical ideas are not bad ideas.
I found the following "commandments" by Robert Schuller highly insightful in determining our organizational behavior tendency to resist radical ideas. If one can relate to experiencing one of these commandments then we have taken the first step in recognizing where to start improving how we approach testing:
Demand change. Once again the approach taken to automating software testing is greatly affected by the environment in which it is expected to perform. Currently, the software quality organization is entrenched in a technician rich paradigm where testing methods are geared toward manual, ad hoc validation of the software driver and associated printer products. This method is often far down-stream from where the eventually discovered defect – actually was introduced in the development design.
(1) The point is that optimization (automation) should be introduced at the most optimal time in the development cycle to prevent defects from being introduced.
(2) To put automation and tools where they are most effective requires the software quality organization to reclassify the roles and responsibilities of its quality engineering staff. Tools engineering should operate closest to our R&D partners in order to provide automation at the most effective time when design defects can be filtered out. This is before driver release candidates arrive in the labs for validation; rather than down stream where canned testing produces limited results.
(3) Software quality must aggressively evaluate the core competencies of its staff and actively anticipate the knowledge base necessary to meet current and future testing requirements for new and emerging technology. The organizations inability to fill requisitions for needed technical staff and FW engineers, for example, is an indicator that internally we have not adequately anticipated demand external to HP; or we have not grown our own competencies internally to meet future need.
(4) Software quality does not employ consistent testing methods that help facilitate automation, practical tools, or overall efficiency improvement. Other side effects to this inconsistency make valuation of cost or time associated with testing nearly impossible. Features may be added, changed, or even removed in development, which tends to ripple through the test plans and relevant test cases. The way in which we characterize and manage our defects is not consistent creating non-value added work activity increasing the cost of testing.
Invent different ways of working.
Here is an analogy to explain the situation in a software quality organization. There are a number of people feverishly bailing water from a leaky boat. The captain brings on more people to help bail but none of them know how to fix the leak in the boat. The weight of the additional people only makes the water rush into the boat faster and now everyone is feverishly bailing out the boat. Meanwhile, the boat has no direction and is going nowhere. The best solution to the captain’s problem might be to bring people on board who can fix the leaks in the boat rather than bail water. Now the captain can stop the leaking, find a direction, and get underway making progress toward a goal .
Our organization must develop a new way of working, by not only investigating but implementing new ways of testing our products. Automated testing methods are one of many ways to do that if properly sponsored and approached. Both management and testing personnel have become accustomed to doing things the same way, with relative success, and continually fall back on inefficient testing methods we are familiar with but cannot make real progress in improving. We have become complacent and indifferent to changing the methods of testing with which we are most familiar and comfortable. So when the work starts rushing in and people become stressed, we all start bailing faster and getting nowhere for the future.
Some common risks to the test automation effort include management and team member’s support fading after not seeing immediate results, especially when resources are needed to test the current release. Demanding schedules will put pressure on the test team, project management and funding management to do what it takes to get the latest release out. The reality is that the next release usually has the same constraints and you'll wish you had the automated testing in place.
Make a contribution every day.
I recommend our organization take a long look at the core competencies of our staff and aggressively promote the learning and growth perspective of current software testing personnel. Development plans should have well defined learning expectations that both managers and personnel recognize as a necessary objective for the success of the individual in the software quality organization. These learning expectations should carry weight in deciding opportunities for advancement, ranking, compensation, and overall employee job satisfaction. We also must maintain a standard of expectations on new personnel that they have the skills to improve our organization now and in providing new ideas to increase testing efficiency for future technology of our products. Any concessions made toward qualifications must be made up through continuing education opportunities. Otherwise, it is easy to see why people become resist to using sophisticated tools and what seem to be complex processes they have done without just fine.
If it doesn’t contribute, it doesn’t leave the garage.
Automating or simplifying unnecessary work only makes it more efficient--it does not make it necessary. Of one of our most recent printer driver products, only 23% of the defects reported resulted in development code changes. Of that 23% resulting in code changes, 21% of those were found by the documented testing cases or "canned tests" that are currently used to develop automated test scripts. Therefore, only 5% of documented or canned test cases result in code changes or product improvements.
Figure 2: Defects resulting in code changes
Tools and automation should be deployed where testing methods have the highest repetition and/or result in the most return on investment – in the form of bona-fide defect resolution that result in product improvements. Moreover, areas of defect prevention in the design phase, code inspection, and validation of feature changes are the best area candidates for organizational and product improvements in software testing.
Believe that together we can do anything.
I believe we can accomplish a great deal more than we currently do. How do we get participation? Here are some suggestions:
Get management commitment to the automation and tooling process
Attention to these requirements will create advantages for the whole organization by improving communication, spreading knowledge, increasing collaborative attitude, and improving motivation and morale.
Invent.
You cannot discover new oceans unless you have the courage to lose sight of the shore.
From a financial and internal business perspective, software quality’s best contribution to the bottom line is in reducing cost. This means focusing our resources and automation tool effort at the most optimal point in the value chain. Tools that will provide verification between R&D delivery of the software driver and preceding planned verification in software quality. Fortunately, efforts are underway to introduce automation where it is most effective. Baseline acceptance, code inspection, driver file and library verification tools are excellent opportunities to reinvent the way we test and deliver our software products.
The learning and growth perspective will require more knowledge workers. To accomplish this requires a regular assessment to increase our internal core competencies and to bring the technical expertise on board that offers efficiency opportunities to the organization.
References:
Resources: