Measuring Integrated Progress on Agile Software Development Projects

Tamara Sulaiman, PMP, CST, SolutionsIQ
Hubert Smits, CST, Rally Software

Summary

Earned Value Management (EVM) is a well known project management technique which measures the integration of technical performance, cost and schedule against planned performance within a given project. The result is a simple set of metrics providing early warnings of performance issues, allowing for timely and appropriate adjustments. In addition, EVM improves the definition of project scope, and provides valuable metrics for communicating progress to stakeholders. The information generated helps to keep the project team focused on making progress.

Despite the benefits that can be gained, EVM techniques they have been slow to be adopted in the private sector, especially in the area of software development. In many industrialized government sectors the acceptance rate is much higher. Many project managers in the private sector have hesitated to use EVM because they perceive the practices hard to implement.

Agile software development methods have been shown to be effective: software is developed faster, with higher quality, and better meeting changing business priorities and changing market conditions. The prevailing wisdom was that EVM techniques were too difficult to implement effectively on an Agile project, and that EVM could not easily cope with changing requirements. The EVM techniques have been adopted for Agile projects, and the correctness of the results has been proven [1]. AgileEVM is a light-weight, and easy to use adaptation of the traditional EVM techniques which provides the benefits of traditional EVM in Agile projects.

Among the benefits of AgileEVM is the ability to use the techniques on simple Agile projects (single team, short duration) as well as on scaled Agile projects (multiple teams at the program or portfolio level). Earned Value and Planned Value calculations for individual teams can be combined for a concise overall picture of the performance of the program against the plan. Because the AgileEVM metrics are expressed the same way that traditional EVM metrics are expressed, a 'roll-up' across hybrid programs (mix of traditional and Agile teams) is also possible.

Why Use Earned Value Management Techniques for Software Development?

It has traditionally been seen as difficult to accurately forecast final project costs for any software development project. The actual performance of a project (teams ahead or behind schedule or budget) cannot be converted easily to a cost forecast. The best option that project managers have is to use notoriously inaccurate estimates of cost of work for this purpose.

In a 1998 article, Fleming and Koppelman [2] assert that "The single most important benefit of employing earned value is the cost efficiency readings it provides...". Earned Value Management has been a recognized project management technique since the 1960's. It is the subject of in-depth study by the Project Management Institute's (PMI) College of Performance Management and is now included as a standard in the "Guide to the Project Management Body of Knowledge" published periodically by the PMI. EVM integrates the areas of technical performance, schedule and actual cost to provide metrics for work actually accomplished. By comparing the earned value (EV) with the planned value (PV) the actual progress on the project is compared against the expected progress which yields valuable information. Using this information, metrics assessing cost efficiency and schedule efficiency are calculated. Metrics forecasting the expected cost to complete a project and total expected cost of a project based on past project performance are derived.

What is Agile and Scrum?

Agile (or lightweight) Software Development Methods have been developing over the last 15 years. They are a response to the often document-centric and heavy processes that are generally used for software development. The core principles of the Agile methods are:

• Iterative and Incremental: software is not developed in a multi-month effort, but in a series of short (2 - 4 week) iterations;
• People centric: the whole team is responsible for planning, designing and delivering the increments. Teams are small, 7 to 10 people and scaling happens by adding teams to the project. Teams are encouraged to be self-organizing;
• Enabling change: Agile methods allow for requirement changes at the end of every iteration, enabling a better tuning of product requirements with the delivery process;
• Business focus: in every iteration, only the features that are of the highest importance for the business are delivered;
• Regular inspection of product and process: at the end of every iteration the delivered product features are demonstrated, and the effect on the product is considered. Also at the end of every iteration the team inspects the way they delivered the features and agrees on process improvements.

A full description of Agile Methods is beyond the scope of this article, we suggest books by Peter Schuh (Integrating Agile Development in the Real World) and Craig Larman (Agile and Iterative Development: A Manager's Guide) as good introductions to the topic.

What do all those new words mean?

This document is heavy on jargon, some often used terms are:

• Agile Methods - The collection of lightweight software development methods that have been developed based on the Agile Manifesto. Examples are Extreme Programming (XP), Scrum and Feature Driven Development.
• Product Owner - The person responsible for the success of the product in the market, and therefore entitled to prioritize the needed features of the product.
• Delivery Team - The group of people responsible for the delivery of the artifacts that together make up the product. They are responsible for delivering the right quality and can therefore determine and estimate the tasks involved in the delivery of the product features.
• Product backlog - a prioritized list of features that make up the product as desired by the product owner.
• Iteration or Sprint - a one to four week period in which the delivery team produces working (accepted) product features.
• Feature - a product component that the product owner and customer recognize and value.

What is AgileEVM?

AgileEVM is an adapted implementation of EVM that uses the Scrum framework artifacts as inputs, uses traditional EVM calculations, and is expressed in traditional EVM metrics. AgileEVM requires a minimal set of input parameters: the actual cost of a project, an estimated product backlog, a release plan that provides information on the number of iterations in the release and the assumed velocity. All estimates can be in hours, story-points, team days or any other consistent estimate of size. The critical factor is that it must be a numerical estimate of some kind. In this article we will use story-points as a measure of story size and velocity.

What does AgileEVM provide that the burndown chart doesn't?

Agile methods do not define how to manage and track costs to evaluate expected Return on Investment information. Therefore the iteration burn-down and burn-up charts (as used in Scrum) do not provide at-a-glance project cost information. Agile metrics neither provide estimates of cost at completion of the release nor cost metrics to support the business when they consider making decisions like changing requirements in a release. AgileEVM does provide this information, and is therefore a excellent extension to the information provided by burn-down charts.

What is our baseline?

We are using three datapoints to establish the initial baseline:

• The number of planned iterations in a release;
• The total number of planned story points in a release;
• The planned budget for the release.

In order to calculate the AgileEVM metrics, there are four measurements needed:

• The total storypoints completed;
• The number of Iterations completed;
• The total Actual Cost;
• The total story points added to or removed from the release plan.

How to calculate Planned Value, Earned Value and Actual Cost?

The comparison of the Earned Value (EV) against the Planned Value (PV) lies at the core of the EVM. Planned Value is the value of the work planned for a certain date. It is the entire budget for work to be completed at the planned date. In Scrum terminology: it is the sum of the estimated feature sizes for all the features up until the planned date.

Earned Value is the value of work completed at the same date as used for PV. Earned Value is not synonymous with actual cost, nor does the term refer to business value. Earned Value refers to technical performance (work) "earned" against the baseline or work planned. In Scrum terminology: it is the sum of the estimated story points for the features up until the calculation date.

Actual Cost is what the name implies: the cost in dollars to complete a set of features.

The reader will be aware by now that the terminology used in this article is specific to EVM. Our day-to-day language has sometimes different interpretations for terms like 'Earned Value'. The article will continue to use the correct EVM vocabulary, as this is important for the use of these metrics in an audience that is familiar with the technique.

An example to clarify these three concepts. With a total project budget of $ 175,000, and having completed one out of four Iterations, we have this product backlog and these actuals:

Feature	Estimate (storypoints)	Completed (storypoints)	Actual Cost (1000s dollars)
Welcome Screen	10	10	15
Advert - Splash Screen	20	20	30
Login Screen	10	10	20
Personalized Google Ads	20
Catalog Browser	20
Catalog Editor	10
Shopping Basket Browser	5
Shopping Card Editor	25
Check-out Process	20
Invoice Calculation	10
Credit Card Verification	10
PayPal Payment Handling	20
Order Confirmation Email	20
Totals	200	40	65

With this information (which should all readily be available in any Scrum project) it is easy to calculate our three base values:

Expected Percent Complete equates to the number of completed iterations divided by the number of planned iterations (in our example, after Iteration 1 we should be at 25% complete);

• Planned Value for a given iteration is the Expected Percent Complete multiplied by the Total Budget (25% of $ 175,000 = $ 43,750);
• Actual Percent Complete equates to the total number of storypoints completed divided by the total number of storypoints planned (40/200 = 20% complete);
• Earned Value is calculated by multiplying Actual Percent Complete by the Total Budget (20% of $ 175,000 = $ 35,000.

AgileEVM works by comparing the current release plan (taking changes in requirements into account) against actual work performed. We can see at a glance that our project is in trouble: the Earned Value is less than the Planned Value (EV = $ 35,000 and PV = $ 43,750).

It is important to note that in AgileEVM there is no credit for partial completion. The backlog items are either done or not done (0 or 100%.) In keeping with Agile terminology: a backlog item is only 'complete' and story points awarded when the customer accepts the item as 'done'.

Analyzing the AgileEVM metrics - using Cost Performance Index and Schedule Performance Index

The Cost Performance Index (CPI) gives a measure of efficiency. It shows how efficiently you are actually spending your budget dollars compared to how efficiently you planned to spend them. It is calculated by dividing Earned Value by the Actual Cost. In our example, CPI is 35,000 / 65,000 = .53.

A CPI of 1 indicates that you are spending your budget to accomplish work at the rate that you had planned to spend it. A CPI less than 1 means that you are over budget i.e. you are spending your budget less efficiently than planned because your Earned Value is larger than your Actual Cost.

CPI > 1	CPI = 1	CPI < 1
Under Budget	On Budget	Over Budget
EV > AC	EV = AC	EV < AC

The Estimate at Complete is the forecast of the total amount that we will need to spend to complete the planned work, based on the actual work accomplished. The easiest calculation is to divide the Total Budget by the Cost Performance Index. While this is not the most accurate calculation, it is accurate enough to identify trends in time to take corrective action. Our Estimate at Complete is $ 175,000 / .53 = $ 330,188 we predict to be 47% over budget.

The Scheduled Performance Index (SPI) compares Earned Value with Planned Value and is calculated by dividing the Planned Value into the Earned Value (our SPI is 35,000 / 43,750 = .8 - we are 20% behind schedule). The analysis of the SPI is comparable to the CPI analysis:

SPI > 1	SPI = 1	SPI < 1
Ahead of Schedule	On Schedule	Behind Schedule
EV > PV	EC = PV	EV < PV

And an Estimated Completion can be estimated by dividing the SPI into the Planned Iterations, in our example we planned to finish the work in 4 Iterations, and expect to need: 4 / .8 = 5 Iterations.

Agile allows me to change the backlog after each iteration, does Agile EVM cope?

Yes, the AgileEVM method does take into account the changes to the backlog that may be introduced after each iteration. By using the changes to the backlog (the added or removed estimated storypoints caused by added or removed features), you effectively 're-baseline' after every iteration. The effect of re-computing the AgileEVM after every iteration, is a validation of the modified backlog against release plan. You are validating that you will be able to complete all of the work planned for the release within both schedule and budget. This gives the Product Owner (who 'owns' the backlog) early information about the effects of his changes, early enough to reconsider changes if the affect the release plan negatively.

How often should Earned Value be measured?

Iteration boundaries are well suited to calculate the AgileEVM, it can be done more often, which is applicable for a release that contains only a few Iterations. EVM has been proven to be statistically accurate as early as when 15% of the planned features are completed, which makes EVM and AgileEVM an important metric to report to your stakeholders. The important consideration is that you have the correct cumulative values available at the boundaries that you choose to calculate your AgileEVM.

How can EVM be 'rolled up' across project teams in a program?

When teams in a program are comparable in size and velocity then the AgileEVM calculations can be made over the whole program. Often teams are not comparable, they work on different parts of the project, with different velocities, different storypoint values and different project costs.

When the latter scenario occurs (teams have different characteristics), calculate the Earned Value for each team, and add them up to a Total Earned Value. The example below shows how we did this for a program with 3 teams. Also calculate the Planned Value for each team and add them up for a Total Planned Value. From there on all of the EVM calculations remain the same, with the caveat that the Actual Cost needs to be the actual cost across teams to calculate a total CPI across a program. The CPI and Estimate at Complete are useful indicators of how the program is doing overall, but does not give you an indication of any particular team that may be in trouble.

Team	Total Budget	Planned Value	Earned Value	Actual Cost	CPI	SPI	Estimate at Complete
Team A	$ 300k	$ 150k	$ 150k	$ 150k	1	1	$ 300k
Team B	$1,000k	$ 575k	$ 500k	$ 625k	.8	.86	$ 1,250k
Team C	$ 800k	$ 175k	$ 200k	$ 180k	1.11	1.14	$ 720k
Program Totals	$ 2,100k	$ 900k	$ 850k	$ 955k	.89	.94	$ 2,360k

In this example, the total program is forecast to be over budget by 11% (1 minus CPI), or $260,000, and is 6% over time. By looking at the results for each team, you can see that Team A is currently performing according to planned cost and schedule. Team B is 20% over planned budget, and 14% behind planned schedule. Team C is performing better than planned, 11% under budget and 14% ahead of schedule.

Conclusion

Using these simple performance metrics in conjunction with the more typical Agile metrics (the Iteration burn-down and release burn-up charts for example) provides objective analysis to share with teams, management and customers. The early warning that the AgileEVM metrics show, validates changes to release plans and provides business with the opportunity to make priority trade-off decisions early in the project lifecycle.

References

1. Agile EVM Research Paper
2. Earned Value Project Management, A Powerful Tool for Software Projects;
   http://www.stsc.hill.af.mil/crosstalk/1998/07/value.asp

Related Agile and Scrum Articles

Agile, Multidisciplinary Teamwork

Adaptative Project Management Using Scrum

Don't Write Another Process

More Scrum Knoweldge

Scrum Expert

Scrum Open Source Tools

Agile Videos And Tutorials

---

Click here to view the complete list of archived articles

This article was originally published in the Fall 2007 issue of Methods & Tools