Going round and round and getting nowhere eXtremely fast?
Another look at incremental and iterative development

Pascal Van Cauwenberghe

‘T was brillig, and the slithy toves Did gyre and gimble in the wabe
From Jabberwocky
"Through The Looking Glass" – Lewis Carrol

Introduction

Surprisingly many people still wonder what’s wrong with the safe and predictable sequential phased approach, also known as "waterfall". This despite the fact that since many years, a lot has been written about iterative and incremental development. Still, most people involved in software could not give you a clear definition of what those two words mean to them. When they are able to give a definition, they often contradict each other.

Into this confused world, the proponents of "Agile Development" and especially Extreme Programming proclaim that they go beyond "traditional" iterative and incremental development methods.

What are we to do? How do we approach software projects? How do we reach our targets reliably, quickly and efficiently? When do we gyre, when do we gimble?

Defining our terms

"When I use a word", Humpty Dumpty said in a rather scornful tone, "it means just what I choose it to mean – neither more nor less".
"Through The Looking Glass" – Lewis Carrol

We must first choose what we want our terms to mean.

A phase = a period in which some clearly defined task is performed. Just imagine we require the following phases: Analysis, Design, Coding, Testing and Integration & Deployment. Each phase has some inputs and some outputs.

To iterate = to perform some task in multiple passes. Each pass improves the result, until the result is "finished". Also called "rework".

An iteration = a period in which a number of predefined tasks are performed. The results are evaluated to feed back to the next iteration.

An increment = a period after which some part of functionality is completed to production quality level.

A release = some coherent set of completed functionalities that is useful and useable to the intended users of the system.

With these definitions we can have a look at different ways to organize a software project.

Different ways to plan a project

The waterfall – straight to the target

The sequential, phased approach, better known as "Waterfall" is the simplest and potentially fastest approach: we analyze the problem, design the solution, implement the code, test the code (are we allowed to fix the coding bugs?), integrate if necessary and deploy. Done.

The attraction of this process model is its simplicity: you do each of the steps and then you’re done. If you want to know how long it takes, just add up the time required for each phase. These times are just estimates, so your total is also an estimate. But it makes following up on the project’s schedule easier. Once, for example, analysis and design are done, they are done.

The simplicity of this model attracts two kinds of people:

Remote stakeholders like upper level managers or end-users who have no need to know the exact process. They really just want to know when the outputs will be delivered.
Teachers and educators who need to present a simplified model of a process as a first, simplified introduction to the subject. Unfortunately, it is this first approximation that sticks in the minds. Or do students skip all the following lectures?

The spiral – iterate to incorporate feedback

Of course, it’s never as simple as the waterfall process suggests: we do make errors during analysis and design. When Walker Royce originally described his process, there were feedback loops back to analysis and design. They were quickly "simplified" away. So, this was the original phased, iterative approach:

Perform analysis, until you think you are ready
Design, until you think you are ready. Should you find any problems in the analysis, feed them back to the analysis. Perform another iteration of the analysis, to improve it.
Code the design, until ready. Should you find any problems in the design, feed them back. Perform another iteration of the design, to improve it.
Test the code. Should you find any problems, start another iteration of the coding.
Integrate and deploy. Feed back up any problems you encounter.

In short: you go sequentially through the phases. Some phases have more than one iteration, if we see that the output of that phase is not perfect. In more than one sense, the waterfall is de optimist’s interpretation of the spiral model.

As Boehm noted, the later a defect is caught, the higher the cost to fix it. We are therefore encouraged to only proceed cautiously to the next phase if the previous phase is relatively complete. As software engineers have noted, the likelihood of finding defects becomes higher as we come to the later phases. This is because each phase delivers a more precise, more "testable" output that the previous phases.

There is one management problem with the spiral model: how do you schedule the iterations? How many iterations will be needed? This makes estimating and tracking the project’s progress more difficult and unpredictable.

Iterative development – getting better all the time

Where the spiral process views feedback and iteration as exceptional events, iterative development assumes we will make mistakes, we will need feedback, we will need several iterations before we get it right.

In iterative, phased development we design our project plan around the idea of iterations and fit the phases in them. For example, we can plan to have five iterations:

In the first iteration we will do 90% analysis, 10% design
In the second iteration we will do 30% analysis, 50% design, 20% coding
In the third iteration we will do 10% analysis, 30% design, 70% coding
In the fourth iteration we will do 10% design, 50% coding, 40% testing and bug fixing
In the fifth iteration we will do 50% testing and bug fixing, 30% integration, 20% deployment.

This process takes into account the reality that we almost never get anything completely right the first time. At first, we will get a lot of feedback. Gradually, with each iteration, the output from each phase stabilizes and we need to do less rework.

Iterative development with incremental delivery - growing software

One complaint we could have with the previous process, is that it takes so long to see the result of the work. Indeed, the software isn’t ready until the absolute end of the project. What if we need the results faster? We could use iterative development with incremental delivery, a variation on the previous process.

The basic idea is to release the software several times to its users, each time with more functionality. Each release is complete, useable and useful to its users. Each release adds more functionality, preferably the most important functionality first. How do we schedule our project?

First, we go through several iterations of analysis and design, until we are pretty sure these phases are mostly complete. We can then select the content and schedule of the increments to be developed. Let’s say we have identified three increments A, B and C, to be delivered in this order to the users.
For increment A, we iterate through all required phases, until the software can be released. We expect to rework the analysis and design very little.
For increment B, we iterate through all required phases, until the software can be released. We expect almost no analysis or design work.
For increment C, we iterate through all required phases, until the software can be released. We expect almost no analysis or design work.

We can picture this process as four applications of the above process. The amount of analysis and (architectural) design decreases in each step. The process delivers functionality after the second, third and fourth steps.

How do we select the contents of the increments? Architecture-driven processes would select the elements that have the most impact on the architecture (and thus the design). Risk-driven processes would select the most risky elements. Users would prefer the most useful or urgent functionalities.

Agile software development – Extremely incremental

Agile software methods like "Extreme Programming" take another approach, based on the following assumptions:

You should welcome feedback and rework, in stead of trying to avoid it
You should deliver software to your users in small increments and deliver functionalities in the order that the users want.

The development process is then scheduled as follows:

Decide on the length of incremental releases. All increments are the same length. Decide, if necessary, which ones will be "internal releases", only used internally for evaluation and feedback, and "external releases", which are released to the end-users.
Gather an initial list of requirements. New requirements can be added at any time.
Before the start of each increment: users (or their representative) prioritize the requirements and assign the requirements to releases. As many releases as required can be planned, but typically the contents of all releases, except the one being implemented, are subject to change.
During the increment, the requirements are analyzed and broken down into small micro-increments. These increments are small enough to be fully implemented in a day or less.
To implement a micro-increment, the developers analyze, design code, test, integrate and deploy iteratively. They iterate very quickly, so as to get as much as possible concrete feedback to guide their work.
Each increment has a fixed length. If needed, scope is reduced (by the user) to meet the target date.

What’s so extreme in "Extreme Programming" ?

What’s the difference between Extreme Programming style planning and iterative development with incremental delivery, such as is possible with the well-known and accepted "Unified Process" framework?

The assumption that you can do architecture and design incrementally. Indeed, XP spends very little time upfront to define an architecture and overall design. Instead, we immediately start with the first increment to deliver functionality. All architecture and design work is done to satisfy the requirements of the current increment. XP does design in the following ways:

A System Metaphor describes an overall architecture and vocabulary that all participants understand and agree with. The Metaphor is refined iteratively over the whole project.

Test-First programming designs the code incrementally by defining executable tests that the program unit must comply with.

Refactoring reworks the design to iteratively adapt the design to changing knowledge, environment or requirements.
Whiteboard design sessions, CRC card sessions… and other group design techniques.

The even more controversial assumption that you can do analysis incrementally. At the start of each increment, we only examine the requirements that have been scheduled for this increment. XP does analysis in the following ways:

Brief requirements, called Stories, are elaborated interactively between the development team and the story author.
Requirements are formalized into executable "Acceptance" tests, which specify and verify the compliance of the software with the requirement.
Requirements are allocated to releases using the Planning Game, where developers and customers optimize delivered value versus development cost.

The assumption that you can deliver software features incrementally, in whatever order yields most benefit for the customer. If the software can be delivered incrementally, the users will get the benefit of the functionality sooner and thus get a faster return on their investment. As the customers can choose the delivery order, they receive the most important and valuable features first. The use of these increments generates lots of valuable feedback to drive the following increments.

What’s the best way to plan a project?

The short answer: it depends. It depends on your project, your team, and your environment.

If the assumptions of XP hold, we can gain a lot of benefits from working incrementally:

Early delivery of useful software.
Delivery of functionality based on the value users give to that functionality (business value driven instead of architecture or risk driven).
Clear, tangible feedback on progress: each increment delivers finished, production quality functionalities.
Clear and regular feedback on the quality and fit of the software from the users to the development team.
The ability to adapt the project plan.
Clear scheduling and predictable delivery
The development of the simplest possible system that satisfies the requirements, without any unnecessary adornments and complexity.

What are the dangers of this approach?

By only looking at small pieces (increments) of the software, we may have to do massive rework, which might have been avoided if we analyzed or designed the whole system correctly.
We might "paint ourselves in a corner": get a system which is incapable of supporting some new requirement.
We may be unable to add "global" requirements, such as globalization, performance, and security… A typical example is trying to retrofit "security" to an application that was not designed with this requirement in mind.
We might go slower because we have to restart analysis and design sessions for each increment, instead of doing it once at the start of the project.

So, it’s not so easy to define how we should organize our projects.

A few heuristics

If your requirements are volatile or incomplete, if the environment changes or if you wish to be able to anticipate to future events, work incrementally. If the requirements and environment are stable, you can optimize your process by looking at all the requirements upfront. When we use an investment analogy, we can compare agile scheduling with options, where we pay a small investment to be able to postpone our buying/selling decisions, versus shares, where we make the full investment upfront. The potential gains from shares are higher (because we don’t pay the price of the option), but the risk is also higher.

If you are not familiar with the domain, haven’t built any similar architecture, work incrementally. Use the feedback to design your system and to avoid unnecessary complexity. If you’ve done lots of similar systems, you can do more upfront. Still, you can always use the feedback to keep your system simple.

If you can’t deliver incrementally, don’t deliver incrementally. But, even if you can’t deliver incrementally to your final user, you might deliver incrementally to internal users, such as QA, product managers… And even if you can’t do that, deliver incrementally within your team. That way you can still get the benefit of early feedback.

If you know a requirement is going to be more difficult to work on if you only tackle it later, work on it now. This can happen with "global" properties of software systems like localization, scalability, and security. You can still get the benefit of business value driven scheduling for most requirements if you take those few exceptions into account: "Did you want security with that, Sir? With security, this feature will cost you X. Without, it will cost you Y. If you wish to add security later, it will cost you Z." Iterative (re) design techniques like refactoring and the emphasis on simplicity does help to keep your software "soft", more malleable, more accepting of changes.

If your team is big, do enough work upfront with a small team so that you can decompose the work on the system on the basis of a reasonable (but not perfect or final) architecture.

My approach

I prefer to err on the side of doing too little work upfront, rather than too much. That’s because I’ve learned some things by applying Agile processes:

If you use simple designs and keep them simple by refactoring you can keep your software so changeable that you can accept most requirements without much trouble. On a well-functioning XP team it feels as if you could go on forever adding features, if only the users could keep coming up with them.
If you are afraid to change something, because you fear the refactoring will be a lot of work and break things, do it often. For example, in my team we’ve learned to do database refactorings often, updating our schema every few days, with a minimum of work and without our users noticing.
No matter how well you know the domain or the architecture, you will learn new insights from feedback. The most important thing I’ve learned is to write code that is a lot simpler than it used to be.
Customers are uneasy with the apparently complex or chaotic scheduling of iterative and incremental processes. They much prefer the safety and simplicity of a waterfall process. If you can involve them and have intermediate incremental deliveries, within the overall "waterfall", they quickly appreciate the feedback and steering power that you give them. And they will never work any other way again.
Keep your team as small as possible. A smaller the team requires less communication overhead, less upfront planning and decomposition and gives more agility to respond to changes.
The assumptions of XP might not hold completely, but often it’s close enough. For example we can assume that most requirements can be analysed, planned and implemented relatively independently so that users are free to schedule them as they see fit. We can handle those exceptions, where there are dependencies or risks, appropriately, so that we have the best possible schedule from both a functional and technical standpoint.

Conclusion

Incremental software development, with short increments, is a very useful technique to reduce project risk and to deliver value faster to users. Each release provides opportunities for useful feedback and replanning, so that the final product meets the users’ real requirements. The project is at its most "agile" when we consider only one increment at a time.

We must balance this with some forward-looking actions, to avoid some pitfalls and to go faster. We can plan and implement several increments or perform some upfront analysis and design before starting the increments.

When shouldn’t we work completely incrementally?

When we have a lot of knowledge and experience in the domain, the architecture and the environment. We can "invest" our knowledge to enable us to take the "best road" to our destination.
We think the cost of implementing the requirement will go up sharply as the project progresses.

There are a few heuristics and you should discover more. You must choose the right combinations, based on your experience, your environment, and your task. And you must always be ready to learn from feedback and to adapt your plan and your process accordingly.

Related Methods & Tools articles

Agile, Multidisciplinary Teamwork

Opening Communication within a Scrum Team

Scrum Books

Coaching Agile Teams

Succeeding with Agile

More Scrum Knoweldge

Scrum Open Source Tools

Scrum Expert

Scrum Articles

Agile Software Development Portal

Agile Videos And Tutorials

Back to the archive list