---

Personal Quality Management 
with the Personal Software Process - Page 2

Alan S. Koch, ASK Process

Results in the Classroom & Field

Personal Quality Management is a significant component of the PSP. A growing body of evidence shows that the PSP significantly improves the engineers' ability to remove defects early without compromising their productivity.

The SEI published a study of the data from 298 PSP students [Hays] that showed the differences in programmers' performance as they progressed through the class. Among other things, they removed more than five times as many defects before testing by the end of the course than at the beginning,

Companies that have adopted PSP have confirmed that these dramatic improvements can be achieved in business. Teradyne, Advanced Information Services, Hill Airforce Base and Harris Communications have publicly reported dramatic results. (See my web site at www.ASKProcess.com for a relatively complete compilation of published results.)

Boeing reported the most striking among the results at the 2000 SEPG Conference, when they compared their first project by PSP-trained engineers with the prior three projects from the same group. In spite of being significantly more than twice as big as the prior releases, system test was complete in less than a week, as compared with a month or more for the others. Boeing attributes this 94% reduction in test time to the 75% reduction in the number of defects that were found in system test. PSP's early defect removal paid significant dividends.

Although there is significant variation from one organization or individual to the next, the available data suggest that reasonable expectations for software engineers would be to remove 75% of all defects before integration without extending the project schedule, followed by significant time savings during integration and system testing. And all of this is achieved while enhancing the professional satisfaction (and hopefully, recognition) of the software engineers.

Principles and Practice

Personal Quality Management is not a single thing that you can do. It consists of a number of interrelated activities that build upon each other, resulting in the dramatic improvements we just reviewed. The remainder of this article will examine these specific activities.

Personal Reviews

There is a popular belief that a software engineer can not review his or her own code. The PSP proves that this belief is false if the reviews are done correctly.

An effective code review is not just a matter of reading the code and hoping to find the problems that are hidden in it. Like many other activities, personal reviews are most effective when they are based on a well-defined and structured process. An engineer's personal review process must be structured to direct his or her attention toward the types of defects that he or she has historically injected. The PSP teaches engineers to scan their code multiple times, each time looking for a specific type of defect. This method of focused scanning is remarkably effective at finding defects, and is also quite efficient.

I had been doing personal reviews by simply reading my code for several years, but was finding fewer than 30% of the defects I had injected. After learning the PSP methods, my results were similar to those of many other people, with my personal reviews finding more than 75% of my defects! PSP-trained engineers will agree that personal reviews are definitely worthwhile!

Defect Logging

As was mentioned above, personal reviews will only be effective if they are personalized so they direct your attention to the defects you normally inject. This presupposes that you know your own defect history. As with anything else, informal memory is not likely to serve you well in understanding your defect history.

There is only one way to construct an actuate defect profile for your work: Log every defect that escapes the phase in which it was injected. The PSP suggests logging the following information for each defect:

• Type - To manage the large volume of defect data that you will quickly amass, you will need a way to categorize and summarize it. The next section will describe this step.
• Phases injected and removed - By identifying when each defect was removed and estimating when it was injected, you can understand which defect detection activities are working well and which need to be improved. (The phases identified by the PSP are Design, Design Review, Code, Code Review, Compile, Unit Test and After Development.)
• Fix Time - The number of minutes required to remove the defect (including research, debugging, reworking the design, reworking the code, compiling and re-testing) allows you to understand the magnitude of the problem posed by each defect.
• Fix Defect - If the defect was injected by fixing a defect, record a reference to the other defect so you can understand the types of validation that defect fixes require.
• Description - A concise description of what was wrong (not the symptom, but the cause) will allow you to recognize similar defects in the future.

Making a defect log entry takes only a few seconds, so it has little impact on your productivity. However, if you fail to log the information immediately, you are unlikely to be able to remember all of the defects and the data for each one. This information is critical to your ability to perform effective personal reviews, and defect prevention (which we will discuss later).

Defect Categorization and Analysis

Categorizing defects allows you to group them for convenient analysis. Without reasonable categories, your defect data will be a mass of data from which it would be hard to glean patterns. The PSP suggests a preliminary categorization scheme, but it also teaches engineers to modify that scheme based on their actual defect data.

Pareto analysis of categories helps each engineer to identify actions that are warranted based on his or her own data. For example:

• Re-working the defect categories may be indicated if there is:
• An ill-focused category into which many defects fall or
• A number of closely related categories that each has very few defects;
• Changes to your processes may be indicated in order to catch defect types that:
• Occur quite often or
• Require a lot of effort to remove.

The changes to your processes that can target specific defect types fall into two main categories:

• Early defect removal - Changes to your review (or other defect detection) processes can be designed to improve their effectiveness at detecting specific defect types, or
• Defect prevention - Changes to your design or coding processes can eliminate certain defect types. We will discuss this next.

Defect Prevention

Often, when we identify problems with the way we do our jobs, our first reaction is to resolve to try harder or work more carefully. While such resolve may produce some immediate improvement, in the long run just trying harder will not result in lasting significant change. As soon as other issues attract our attention, our resolve will refocus on those issues, and the initial improvements are likely to slip away.

Preventing defects requires that we make changes to the processes we use to do the job. Those processes then provide the continuous and unwavering resolve we need to make the improvements permanent, and prevent the problem we are attempting to eliminate from returning.

Once you have identified a defect type that is a candidate for prevention, you might eliminate it be doing something like this:

1. Focus on one particularly troubling type of defect. Trying to attack several problems at once is likely to doom your effort to failure. By attacking one single important issue at a time, you can maintain the focus that is more likely to result in success.
2. Identify the conditions under which that defect type occurs. Prevention requires that you understand why you inject that particular type of defect. When does it happen? Under what conditions? What normally are the circumstances at the time? Remember that defects are the result of your human errors. So we are searching for the root cause of those errors.
3. Consider how to change the process you use. Your objective is to eliminate the defects that result from your errors. You can accomplish this in one of two ways:
• Eliminate the error. For example, I eliminated begin/end mismatches by changing my coding process: I now write "Begin" and "End" at the same time, then go back and fill in the code between them.
• Change the situation so that your error does not result in defects. For example, you might avoid the consequences of forgetting to declare variables by using an editor that highlights undeclared variables as you type (much like the spelling checker in Word).
4. Test the process change and collect data. Don't just assume that the changes you decide on will be effective. Try them out on at least one pilot project and collect the necessary data to understand if they are effective. If they are not effective, return to step 3 with the new information you collected from the pilot test to try to develop a better solution.
5. If the process change is effective, permanently adopt it. Change your written process description, checklists or other mechanisms you may use to assure that you follow your new process. (If you have never written your process steps down, doing so may be your first step to avoiding errors!)

The sources of problems among different software engineers are as varied as their personalities. But from my experience teaching the PSP, I have discovered that for many software engineers, a highly effective way to prevent a variety of defects is to formalize the design process and/or design notation. Most of us spend insufficient time working through the details of our program designs, and this results in some of our most expensive and frustrating defects. The PSP recommends a set of design templates that will assure that all of the important parts of the design have been examined.

While defect prevention provides the greatest benefit, the reality is that we will not be able to prevent most of our defects. As we said earlier, because we are human we are prone to error, and those errors naturally result in defects. For this reason, the rest of this article will focus on ways to detect defects earlier in the development process, when they are less expensive to remove.

Review Checklists

Focused checklists are the single most important tool you can use to assure the effectiveness of both personal and peer reviews. While the review process itself will assure that you do the requisite things, only a checklist can ensure that you don't miss any important detail. Peer reviews will be discussed in the next section, so the remainder of this section will focus on checklists for personal reviews.

To get the maximum benefit from your personal reviews, your checklists must be focused and personalized:

• Your checklists must check for the defects you normally inject. That means that your checklists must be based on your personal defect history, and not on other people's performance or your organization's norms. Your checklist should check for compliance with organizational standards only to the extent that you violate them.
• Each of your reviews (high-level design, detailed design, code) needs a different checklist. Each checklist should include only the defects that you normally inject in the immediately preceding phase. Do not check for defects that escape prior reviews. (If prior reviews are consistently missing certain defects then fix those reviews.)
• If you use multiple programming languages, then you need different detailed design and code review checklists for each language. Although there is likely to be some overlap or duplication between checklists for different languages, it is important that each checklist fully and exclusively support the reviews for which you use it.
• Your choice of items for each checklist must be based upon Pareto or other analysis of your defect data. It would be infeasible to check for every possible defect. To make the best use of your review time, make sure you are checking for the defects that you are most likely to have injected. It is best to limit each checklist to one page. This means that you will be able to check for no more than 50 or so items in each review.

Just having a checklist doesn't automatically improve your review effectiveness. You must learn to use it systematically. For example, most people find that their reviews are most effective when they scan the work product for one checklist item at a time, rather than reading through the work product once looking for all checklist items. While it may seem that scanning through the work product multiple times is wasteful, this turns out not to be the case, because when you are looking for only one thing, you can scan at a relatively high rate.

The most important thing to keep in mind when doing personal reviews is to avoid rushing. The PSP teaches engineers to review code no faster than 200 LOC per hour. Most people's data shows that when review rates are higher than this, effectiveness becomes spotty, and more defects escape the reviews. Your objective in the reviews is to assure yourself that the defects for which you are checking are actually absent from the work products. The review must be careful enough so that checking off a checklist item is a personal certification that the defect type it represents is absent from the work product.

The final step in assuring the effectiveness of your reviews is to periodically study your defect data to see what it says. Look for types of defects that are consistently escaping your reviews, then consider what it may be about your review process or checklist that is allowing this to happen. For example:

• The defect type may not be represented on your checklist, suggesting that it should be added. If your checklist is already a full page long, this would mean deciding which checklist item must be dropped to accommodate a new one. Naturally, you should drop one that never or rarely results in identifying defects.
• If your checklist includes an item that covers this defect type, then you may want to re-word the item or split it into two or more items so that your attention is focused more precisely during reviews.
• Finally, the fault may lie with the way you use the checklist. A change to your review process may provide the improvement you need.

The purpose of your review checklist is to provide the focus that will assure that the time you spend doing reviews is effective in improving the quality of the work products you produce.

Peer Reviews and Inspections

If your organization consistently does peer reviews of technical work, then you should make sure that you take full advantage of them. While personal reviews can find up to 80% of the defects in your work products, the time spent in peer reviews is still very much worth while. By combining effective peer reviews with effective personal reviews, you can remove more that 90% of the defects from your software before the first test!

Peer reviews can take a wide variety of forms. The most representative of those forms are these:

• Walk-throughs involve the author walking through the work product and explaining what was done and why before a group of peers. The peers interject questions about the work product to understand it better and to identify defects. These reviews require the least time of any of the peer review types. However, they tend to be a rather poor vehicle for identifying defects, being more valuable for educational purposes and assuring that team members understand each other's work.
• Full reviews involve peers examining the work product independently, then presenting questions, concerns and defects to the author in some way. The feedback mechanism can be as informal as e-mail or as formal as a walk-through meeting. The problem with this review method usually lies in the reviewers spending enough time in independent examination. Even when a formal feedback meeting is held, peers will often have spent little or no independent time on the work product. The effectiveness of these reviews can be highly variable, and often depends on the level of stress in the organization.
• Inspections (Often called "Fagan Inspections") are the most formal of the peer review types. Participants receive training, are assigned formal roles and are required to fully participate in the process. Inspections are generally checklist-based, with each person spending time in independent examination. The formal meeting is not held unless all participants have completed their examination of the work product. The inspection meeting includes a reader (not the author) walking the reviewers through the work product, with the reviewers commenting or questioning on each part. Questions that can not be quickly resolved in the meeting are assigned to individuals for resolution, and all identified defects are entered into a defect tracking system to assure they are corrected.

The most effective of the various review forms tends to be formal inspections. Although they are more work to establish and require more of people's time to do, they are also the most effective at finding defects, so they result in lower over-all cost of quality. Even if your organization does not have a formal peer review program, you can still capitalize on this powerful tool by asking your peers for their opinions on an informal basis. Obviously, you can not do formal inspections this way, but you can ask your peers to do independent examination and even attend review meetings. Often these sorts of informal reviews result in the desire within the organization to institute a more regular and formal peer review program.

© 2000-2007 Copyright ASK Process, Inc.

Page 1    Page 3    Back to the archive list