Click here to view the complete list of archived articles
This article was originally published in the
Winter 2004 issue of Methods & Tools
Click
here to reach PDF area
Adaptive Project Management Using Scrum
Craig Murphy, www.craigmurphy.com
Scrum is a lightweight process that can manage and control software and product development: it is a project management process. However, instead of promoting the traditional analysis, design, code, test, deploy "waterfall" approach, Scrum embraces iterative and incremental practices. Similarly, instead of being "artifact-driven", whereby large requirements documents, analysis specifications, design documents, etc. are created, Scrum requires very few artifacts. It concentrates on what’s important: managing a project or writing software that produces business value.
What is Scrum?
The first question that you’ll probably want answered is: is Scrum an acronym for something? You would not be alone in thinking this; most other project management processes and techniques are acronyms. Scrum however is just a name, it’s not an acronym. Similarly, it has very little to do with rugby either, although there is a "daily meeting" that might be compared to a traditional rugby scrum.
Scrum borrows from the "agile" world through its promotion of iterative and incremental practices. It has a simple implementation that is designed to increase productivity and reduce the time it takes to benefit from a software/product development. Importantly, it embraces adaptive and empirical systems development.
That last paragraph introduced two of Scrum’s key facets: adaptive and empirical. The majority of project management methods and techniques are very prescriptive; they tie us down to a fixed sequence of events and offer little in the way of flexibility. Similarly, newer, less mature methods often claim to be a panacea, yet they lack any definitive proof or track history. Fortunately, Scrum is mature; it has a track record going back as far as 1995 and earlier. Similarly, it is scaleable: Scrum can be used on projects of any size. Whilst I will not be discussing it in this article, "Scrum teams" allow Scrum to be used to manage enterprise-level projects.
Scrum also promotes and lends itself to managing eXtreme Programming (XP) based projects. XP uses "customer on site" as a means of ensuring that the development team’s questions are answered but also to ensure that what the development team is producing what the customer actually wants.
The Opposite of Waterfall
Traditionally, we followed a cycle involving requirements gathering, analysis, design, develop, test, deploy with each stage being completed before moving on. This was known as the waterfall approach. Whilst there is a place for the waterfall approach, it has been the subject of a torrent of abuse in recent years. Most notably, the waterfall approach promotes the creation of up-front documentation before any real business value is created. This is confounded by the fact that product development is started downstream, or much later in the project’s expected timeframe. This has the obvious disadvantage of delaying the point at which business value can be realised.
But it gets worse: the customer/user endeavour to ensure that all of their requirements are documented during the early stages, thus the feature set is top-heavy. Failure to prioritise the feature-set often results in low quality systems that are overloaded with features that the customer/user does not actually require in "version 1". Indeed followers of Mary Poppendieck’s work (www.poppendieck.com) will know that "80% of a product’s value comes from 20% of its features".
With this in mind, can we conceivably build a product (a software product) that provides 20% of the feature set? Yes, we can and we do it iteratively. We can deliver "version 1" with 20% of the features, then, a little later, "version 2" with a further collection of features. The beauty of this approach is that development of 20% of the features should not take 100% of the project’s expected schedule and budget: we can realise business value much earlier in the cycle.
Scrum embraces the opposite of the waterfall approach whereby we start working on the analysis as soon as we have some requirements, as soon as we have some analysis we start working on the design, and so on. In other words, we work on small pieces at a time. This approach can be called iterative. Each iteration consists of some requirements gathering, some analysis, some design, some development and some testing culminating in an iterative release cycle (many deployments).
Scrum Basics
Scrum revolves around the ethos of simplicity, resulting in delivery of something that moves the project forward. It achieves this by proposing the following questions (referred to as Scrum’s three questions).
|
Scrum asks… |
Fundamental Project Management issue |
|
What have you done during the last 24 hours? |
This is progress, it’s work completed to date |
|
What do you plan to do in the next 24 hours? |
This is forward planning, it is work you are about to do |
|
What’s stopping you getting on with the work of the next 24 hours? |
These are your impediments or obstructions, it might be things you need in order to work… more forward planning. It’s also identification of immediate risks. |
Scrum Roles
Scrum uses three "roles": Product Owner, ScrumMaster and Project Team.
The Product Owner is possibly a Product Manager or Project Sponsor, a member of Marketing or an Internal Customer.
The ScrumMaster is key, he or she "represents management to the project". Such a role usually filled by a Project Manager or Team Leader. They are responsible for enacting Scrum values and practices (more about these shortly). Their main job is to remove impediments, i.e. project issues that might slow down or stop activity that moves the project forward.
The Project Team should consist of between 5-10 members. The team itself should be cross-functional, involving individuals from a multitude of disciplines: QA, Programmers, UI Designers, etc.
The Process
"Out of the box" Scrum is best described by Figure 1. Most projects have a list of requirements (type of system, planning items, type of application, development environment, user considerations, etc.) Scrum records requirements in a Product Backlog. Requirements need not be precise nor do they need to be described fully. As with most projects, the requirements are sourced from the expected users or "the business". The Product Owner prioritises the Product Backlog: items of importance to the project/business, i.e. those items that add immediate and significant business value, are bubbled up to the top.
The Project Team responsible for doing the actual work then creates a Sprint Backlog: this comprises of Product Backlog items that they believe can be completed within a 30 day period. The Project Team may liaise with the Product Owner and others in order to expand item(s) on the Sprint Backlog. After 30 days have elapsed, the team should have a "potentially shippable product increment". I will discuss the make-up of the Project Team later in this article, under the topic: Scrum Roles.
The Product Owner, the ScrumMaster and the Project Team will make an initial pass over the Product Backlog items where they work out roughly how long each item will take. Initially, these are estimates, best guesses. As time progresses, well within 30 days, we’ll know if the estimate was even close.
Scrum lets us refine our estimates on-the-fly: if we believe that a task will take longer than envisaged, we have the ability to say so before the tasks starts. By only ever working with small work packages (time-boxed to 30 days), any schedule/requirement issues are dealt with as soon as they are identified, not much further downstream where the cost of recovery is considerably higher.
What does this "potentially shippable product increment" actually mean? Put simply, every 30 days, the team should provide something of value to the business, something they can use or something that provides considerable direction.
The beauty of the 30 days approach is this: if the Product Owner, customer or the business likes what they see at the 30 day interval, they can then re-prioritise the Product Backlog. This is important as it means that we are only ever producing goods that will be used by the business: remember that only 20% of a [software] product‘s features are used frequently. In other words, 80% of a software product’s "value" comes from 20% of it features.
Figure 1 - The Scrum Process
Daily Scrum Meeting
One of Scrum’s primary practices is the 24 hour cycle shown in figure 1: the Daily Scrum meeting.
How many of you attend meetings on a regular basis? If you are in the corporate environment, meetings seem to be the norm and often they have little business value apart from to act as a caffeine injection mechanism. Folks who sit down at meetings get too comfortable: they attend meetings for the coffee and doughnuts, not for the project’s sake. Once relaxed, those same folks often stray off the meeting agenda and start discussing items that are either on the project periphery or are not even project related. I am sure you’ve all been in such meetings…
Scrum resolves these issues using two simple approaches.
Firstly, Scrum-managed project meeting rooms are devoid of chairs. Attendees have to stand up. This might sound cruel, but it focuses the mind and those folks who are capable of sitting in (often unproductive) meetings for hours on end are soon discouraged.
Secondly, Scrum-managed meetings are time-boxed or time limited. The Daily Scrum meeting is typically time-boxed to 15 minutes. Only extraordinary projects should require more than 15 minutes. Here is the crux: if you can’t say what you have to say succinctly in a short space of time, you’re waffling, get your coat. Keeping it time-boxed focuses folks’ minds and helps keep agenda items targeted at what’s important: Moving the project forward towards delivery of "something"… and identifying and removing obstacles that prevent this goal being met.
The purpose of the Daily Scrum meeting is to answer Scrum’s three questions:
But there is more! The ScrumMaster and the Project Team are the only people who are allowed to talk: outsiders may listen in, but are removed or silenced should they say anything. This is all about who is committed to the project or not. Outsiders tend to volunteer items that are important to them, but not necessarily to the project and its immediate goal: delivery business value within the 30 day sprint.
Scrum refers to those who are committed to a project as "pigs" and those who are not as "chickens". The story goes like this: A chicken and a pig where chatting about setting about a business together. The pig asked the chicken: "What kind of business would we set up?" The chicken thought for a moment and said "How about a restaurant?" The pig liked the idea but asked "What would we serve?" The chicken responded "How about ham’n’eggs?" At which point the pig refused to take the business venture any further. The chicken was confused and asked "Why?" The pig responded "Well, you would only be involved, whereas I would be committed."
One thing I should mention is the fact that Scrum expects late-comers to any of its meetings to pay a nominal £1, $1, or €1 fine: at various milestones within the project the fines are donated to charity. You may find this slightly humorous; however it does focus the mind and it does prevent meetings dragging on because of late-comers (we have all been in meetings that have to start over because of late-comers, right?) Late-comers are simply told to pay their fine, the meeting continues without further ado.
Scrum Artifacts
Scrum has remarkably few artifacts. There are three artifacts, each of which can be managed using nothing more than an Excel spreadsheet. More advanced / complicated tools exist, many are expensive, web-based or are still under development. Web-based tools are great, but they are not much good if there is no "off-line" operation: I conduct much of my project management (and thinking) in airport lounges where Internet connectivity is a pay-for luxury.
Figure 1 hints at Scrum’s artifacts: there is a Product Backlog and a Sprint Backlog. The Product Backlog is a prioritised list of first cut refinements. I say first cut, because the Product Owner is free to adjust the order in which Product Backlog items are development, they are even free to add new items this is the spirit of "agile".
Graphical feedback that can be printed out and displayed in a public place makes progress reporting very visible. Scrum encourages this kind of reporting and offers Burndown Charts as a means of graphically displaying a project’s progress or not as the case may be. We will take a look at burndown charts later in this article.
I mentioned earlier that Scrum’s artifacts can be managed by nothing more than a spreadsheet. Indeed, the Product Backlog can be represented using an Excel worksheet. Each Sprint Backlog then occupies another sheet within the same workbook.
Figure 2 presents a sample Product Backlog. Keeping with the ethos of Scrum and "agile", you shouldn’t be planning more than one sprint into the future. Indeed, the contents of a Sprint are usually defined during a "Sprint planning meeting".
Figure 2 – A sample Product Backlog
Using the information in Figure 2, we can derive another worksheet, the Sprint Backlog (Sprint 1). Figure 3 presents a sample Sprint Backlog: it contains a list of the things that will be "done" during the Sprint. Each Sprint item has an estimate of how long it should take to complete, usually measured in hours. Looking at figure 3 again, you can see that none of the six tasks have been worked on.
During the Sprint’s 30 day period, the Project Team must update the Sprint Backlog. For example, if BC spent four hours on item 1, Remove user kludge in .dpr file, on Tuesday 2nd (November in this case), he would enter ‘4’ into Sprint day 2’s Backlog item 1 column/row combination.
Figure 3 – Sprint 1 at inception
Keeping the Sprint Backlog updated is key: not only does it allow us to work out how fast a team can work (their velocity), it is an early warning indicator.
What is useful about the Sprint Backlog is its ability to be displayed graphically. Scrum uses burndown charts to represent "work done". Figure 4 presents a burndown chart where no work as been performed in the sprint. The idea behind burndown charts is that they should demonstrate a steady drive to zero hours remaining: it represents a pace of work that should be sustainable. In reality however, some work takes longer than others, and some are even shorter, so the burndown graph may not be a perfect straight line.
Figure 4 – Sprint Burndown, no work performed
Figure 5 presents a Sprint backlog that has been updated after work has been performed. Following the "Remove user kludge in .dpr file" item through, BC performed 4 hours work on Wednesday 3rd, followed by 2 hours work on Thursday 4th and finished the task on Friday by spending 2 hours on it.
You will also notice that on the same Wednesday, BC spent 4 hours on "Remove cMap/cMenu…". Assuming a working day of 8 hours, BC has split his time between two tasks.
Figure 5 – Sprint 1 after work has been performed
The beauty of the Sprint Backlog is its simplicity. By recording the hours of actual work completed versus the estimated hours to complete, we are able to plot equally simple, but powerful Burndown Chart that should provide you with the confidence that progress is being made. If the Burndown Chart does not indicate that progress is being made, then it has served another good purpose: it gives you an early warning that this sprint contains either too much work or too little.
Figure 6 presents a Burndown Chart based of Figure 5’s Sprint Backlog. Ideally, the Burndown Chart should indicate a level velocity, i.e. work is performed at a steady rate. In reality, we need to factor in weekends and other week-day distractions. As we will see shortly, if a Project Team or individual is distracted for a few hours, Burndown Charts are a great way of identifying the distraction very early in the project.
Figure 6 – Sprint Burndown after work has been performed
Figure 7 presents a burndown chart demonstrating that progress is being made, however by no means fast enough. The Burndown Chart gives us a clue that there is something wrong within 2 to 3 days of starting and confirms that work is not progressing at a good speed by days 7 through to 14.
There are a few reasons why this shape of Burndown Chart appears:
Figure 7 – Sprint Burndown after work has been performed, but not fast enough
Of course, Burndown Charts can reveal that the Sprint Backlog might finish early. For example, if a sprint does not contain enough work, you might end up with a burndown chart similar to figure 8. There are other reasons why a Sprint Backlog might appear to finish early:
Excessive working: if the Project Team works more that an 8 hour day (in this case), work might be completed "ahead of schedule". In reality, excessive working only ever appears to have short-term gain – the project will pay for it either in a loss of product quality and/or tiredness downstream resulting in loss of productivity.
Sprint Backlog item estimates may be incorrect: we may have to revisit our initial estimates as the work is being completed well within the existing estimates. This kind of exception can be caught if the Project Team know to announce the fact that the have completed a task ahead of schedule – the Sprint Backlog simply requires that a task is marked as having "0" hours remaining to indicate that it is complete.
Figure 8 – Sprint Burndown after work has been performed, but too fast (not enough work)
In reality of course, we notice burndown charts that have their ups and downs whilst still meeting their target. If the ups or downs are dramatic, then alarm bells can ring indicating that something is wrong with either the sprint’s scope or the sprint’s Project Team. Either way, you’ll get an early warning indicating that this sprint (less than 30 days of work) is about to encounter a problem. It’s better to encounter and solve project issues as soon as they arise: Scrum’s burndown charts put the project issues into context, better that you "lose" 30 days early in the project than you have to find time later in the project to fix mistakes made early on.
Sprint Backlogs and Burndown Charts are early warning indicators, they highlight "lack of progress". Similarly, they highlight scenarios where there is not enough work or work is too easy". However, their use does assume that everybody is committed to keeping the Sprint Backlog up to date and that is a job for the ScrumMaster.
With careful use, even these simple spreadsheets can assist in a number of other ways. For example, it is possible to extract "individual" Burndown Charts for each Project Team member. Whilst an overall Burndown Chart might highlight an overall problem with the project, using individual Project Team member Burndown Charts can help identify the one team member who is causing the problem!
Similarly, by allocating Sprint Backlog items to team members, we can respect their working time: ideally no employee should have to work more than a given number of hours per day (in this case 8).
Why Iterative Development Works
Consider figure 9, it presents an ideal profit and loss curve comprising of cost over time. Early in most projects, there is an initial hit, a start up cost and that’s the period below the line.
After some time passes, typically a number of years, the project starts to make some profit and that’s the period above the line.
Figure 9 - Profit & Loss using "traditional" development (from Software by Numbers)
The crux is getting the project into profit as quickly as possible. Obviously there are a number of tricks that you can use to achieve this, some of which may involve cutting corners, reducing quality, reducing scope etc. Beware if you do use such techniques – early inducement of the profit position point can be a short-term solution that may affect future profit. Poor quality (of product, of support, etc.) can lead to a reduction in future sales.
Iterative development is a technique that allows a controlled inducement of the profit position: if we can release a product earlier on an iterative and incremental basis, we can realise the benefit of the early release much earlier on in the project. Consider figure 10, it presents a typical profit and loss curve where iterative development has been applied.
Figure 10 - Profit & Loss using iterative development (from Software by Numbers)
Iterative development means you are able to deploy your application sooner than before. The earlier you are able to deploy your application, the sooner you can reap the profit.
Scrum promotes iterative development: every 30 days you have a "potentially shippable product increment" (or executable). If you are able to move much of the useful functionality into early releases, not only are your customers going to appreciate early access, they’ll be in a better position to confirm that they’re getting what they asked for.
Adapting Scrum
I have enjoyed reasonable success by adapting Scrum’s three questions. Early in 2004 I used the three questions in front of a small audience: they really appreciated the simplicity. They also appreciated the honesty that the questions seemed to portray: because of their simplicity, it’s difficult to incorporate untruths into the answers to these questions.
I discovered that my upper managers wanted four questions answered:
Ignoring question 1 for now, it’s clear that questions 2, 3 and 4 map directly on to Scrum’s three questions: "what have you done?", "what do you plan to do?" and "what’s getting in your way/stopping you working?".
If we keep track of our responses to Scrum’s first question, "what have you done?", it lets us answer question 1, it gives us a summary of work completed to date. Indeed, whilst I have not been able to implement Daily Stand Up Meetings, because of geographical and diary time issues, their benefit has not gone unnoticed. I have had some buy in to "stand up" meetings, but not enough: we don’t practice them right now.
Summary
I hope this article has provided you with a flavour of what Scrum is and how it can help you manage your next (or even current) project and help you deliver software incrementally instead of "big bang".
Keep asking these questions:
I will let you think about figure 11. Where does your current/future project appear? Scrum lets us move our projects from "top right" towards "bottom left". Those projects that try to implement 100% of the requirements move themselves into anarchy; they try to do too much. By doing less work, accepting that 80% of a product’s value does come from 20% of its features means you can move through complex and complicated towards simple.
Figure 11 - Where’s your project?
Lastly, if you take one thing away from this article, it has to be Scrum’s co-founder, Ken Schwaber being quoted in April 2004, Vienna: "Don’t procrastinate, do something, no matter how small…"
Scrum: the ethos of simplicity and the art of the possible.
This article’s resources