Finding a Partner to Trust: The Agile RFP

Peter Stevens, https://www.scrum-breakfast.com/

Customers of software development services who want to outsource a software development project face a problem: Traditional methods of selecting a software developer are expensive, time consuming and optimize the wrong criteria. They set the stage for delays, cost overruns, and building software with poor or no return on investment.

Agile methods, including Scrum and XP, have proven successful at creating great solutions that meet the expectations of their sponsors and users. Many organizations would like to apply agile approaches, but the traditional tools for selecting a vendor don't mix well with agile development.

By conceiving the project from the beginning as an agile project, you can outsource projects effectively and agilely. This paper:

1. Describes how one team used Scrum to create an agile RFP
2. Discusses what information should be present in an agile RFP
3. Proposes how to find a partner to trust through a lean, Agile selection process

Background

My customer wanted to develop an application to automate critical work and information flows. These are very complicated, involving direct customers, end customers, the company itself and its suppliers. The company had some expensive experiences with waterfall development processes. Scrum had usually been the solution, so for this project, it seemed logical to conceive the project to mix well with Scrum. But this presented new problems: How do you write an agile, Scrum-Compatible RFP? How do you select a company to implement an agile project?

A Google search [1] was remarkably unhelpful with this problem. The top links all pointed to a paper and presentation from 2003 about combining Use Cases and User Stories in the RFP process. A request to the ScrumDevelopment Group produced no responses [2]. So we were on our own.

A Classical RFP is poorly suited to Agile Development

A request for proposal would typically include items like the template in Box 1. The customer seeks to specify his requirements precisely; the supplier should say exactly what they are going to do, how long it will take, and what it will cost. Because requirements, time tables and deliverables are rigidly specified, this form of contracting does not fit well with agile development processes.

Template for a traditional RFP Introduction and Background Purpose of the Request for Proposal Schedule of Events Guidelines for Proposal Preparation Proposal Submission Detailed Response Requirements Executive Summary Scope, Approach, and Methodology Deliverables Project Management Approach Detailed and Itemized Pricing Appendix: References Appendix: Project team staffing Appendix: Company Overview Evaluation Factors for Award Criteria Scope of Work Requirements Deliverables Source: Foundstone RFP Template, http://www.foundstone.com/us/services/foundstone_rfp_template.doc

Using Scrum to create the RFP

My customer assembled a team of two domain experts and myself as Scrum coach to put together the RFP. We decided to use Scrum to create the RFP. During "Sprint Zero", which only lasted a week, we were briefed by the product owner. Then we created and estimated the product backlog. We reserved a meeting room for the duration of the project, which gave us a place for the burn down charts, task board, and other information that gets hung on the wall in a Scrum project.

We considered the RFP itself to be a product, so we used the agile workshops originally defined by Mike Cohn [3] to figure out who our readers are and what they want from the document. The users included:

• Our own management, who need a clear vision of what the product will be and what it means for the company
• Prospective suppliers, who need to understand the business process which shall be automated and what the new system should do.
• "Everybody" who needs to understand how Scrum would be applied in this project.

We came up with a table of contents which included important elements of a classical Project Plan [4]

1. Project Charter

1. What are overriding the corporate goals -- taken from the corporate website
2. Elevator Pitch - what will the product do for its users?
3. Product Mission - what will the product do for the company?

1. Current Situation: How is the problem being solved today?
2. Desired Situation: How should this system solve problem?
3. Risk Analysis: What has gone wrong in the past and how do we want to prevent that in the future. What are the big non-functional problems that have to be solved?
4. Project Management Procedures, Roles and Responsibilities: A description of Scrum and its roles.
5. Budget Recommendation: How much should the project allowed to cost based on the benefit it will bring to the company?

Each entry in the table of contents became a theme for grouping the stories. Each story corresponded to some content in the RFP or an appendix. For example, we had stories like 'Current Process Diagram' or 'Project Management Procedures' which corresponded directly to (sub-) chapters in the RFP.

Sprint Planning and Demos

Each sprint lasted 2 weeks. The Sprint Review and Planning Meetings were not as formally separated as in Scrum by the book. The Product Owner gave us guidance, first in the Kick-Off meeting, later during the Sprint Reviews and the team decided how to translate that guidance into content in the RFP. So the team was truly self organizing and assumed some of the duties of the Product Owner.

Based on our initial estimates, we set the following themes for each Sprint.

1. Project Vision
2. Current State
3. User Interviews
4. Desired State
5. Final Document
6. Retrospective/Next Steps

It didn't quite work out that way. In the Sprint 1 Retrospective, we realized that we should be delivering the RFP incrementally and that we should work toward a definition of done. So we decided that a Story is only finished when the corresponding information has been integrated into the RFP document and when that chapter has been reviewed within the team for content, spelling and grammar, and the updated RFP is posted on the Wiki. After Sprint 3, we could have given the RFP to potential suppliers after any Sprint (perhaps with a little formatting), and they could have starting the bidding the process.

Sometimes we did break stories down 'horizontally', e.g. each user interview was a story, even though a user interview produced no content for the RFP. Such stories had to produce a clearly defined deliverable which was useful for creating the final deliverable. So a user interview story produced a summary of the interview which the team used a basis to discuss the feedback.

The First Sprint

The application should automate the information flow between the company, its customers and its suppliers. The process is very complicated and had been analyzed previously, so we wanted to build any work which was already available.

The first sprint focused on two issues:

1. What is the product charter: Who is the product for? What should it do for them? What should it do for the company?
2. What previous analysis was available?

The first drafts of the Product Mission and Elevator Pitch provoked a lot discussion with the product owner and beyond. Small differences in the wording of either can have tremendous impact on the product, adding or removing many person-months of work. Lack of clarity increases the risk of building the wrong product, not including critical functionality, wasting resources on unneeded functionality or endlessly retargeting the product as management changes its mind about what the vision is.

Using Scrum for the RFP Project

We used classic Scrum to create the RFP. Stories corresponded to chapters or subchapters of the RFP. The stories were estimated in Story Points. At the beginning / end of each sprint we discussed the results with the product owner and planned the "functionality" for the next sprint. Each story in the sprint backlog was broken down into tasks and estimated in hours. We used release and sprint burn down charts to track our progress, occasionally adjusting the scope to ensure that the project would deliver the RFP on time.

We used 'tangible tools' - cards and flip charts - to manage the backlog, tasks and burn down charts.

By the end of Sprint 3, we had a document in close to final form. There is a special moment of recognition in the Product Owner's eyes, when he sees that he will get his product as wants it. People who have worked on a waterfall basis aren't used to experiencing that until much later in the project, if at all. By the last sprint, we were starting to wonder if the additional work we were investing wasn't just 'gold plating', which made it easier to declare the RFP 'Done!'

Contents of an Agile RFP

Agile projects set different priorities than traditional approaches and this has an impact on the contents of the RFP. The most important differences between agile and traditional approaches can be summarized as follows:

	Agile Project	Traditional
Objective of Project	Meet goals and needs of Organization, Users and Customers by providing them functionality which helps them accomplish their goals.	Realize defined scope within defined time and budget
Scope	Negotiable - realize the minimum set of features required to meet the objectives of users and customers	Precisely defined in advance. Often expressed as wish list, which gets cut down in the process of negotiations to meet budgetary goals.
Scope Changes	Embraced. The priority of not yet implemented functions can be downgraded in favor of new requests which are judged more important.	Discouraged. Changes are often a source of delay or cost overruns. Change requests are often used by vendors to restore profitability to projects where fierce competition resulted in unprofitable basic prices.
Time	Release quickly to start generating ROI	Release once with all functionality
Quality	Actively defined and agreed upon through definition of 'done', which is confirmed at the end of every Sprint	Assured in a separate QA-Phase. This increases the risk of delay and cost overruns, because errors detected late are more expensive to find and fix.
Trust	Pre-Requisite for working agilely, difficult to establish before work starts.	Attempt to compensate lack of trust through contractual process, penalties, etc.
Cost	Ideally planned proactively as a function of the value which the project should produce.
Risk	ROI is managed continuously by the product owner. Incremental delivery, regular inspections and prioritization (delivering the most valuable work first) are the primary tools for mitigating risk.	Risk of cost overrun is a major factor in planning the project. Fixed Price/fixed scope, cost ceilings, penalties for late delivery are used to minimize financial risk

Because of the differences, our RFP had a similar structure but different contents than a typical, waterfall oriented RFP:

1. Introduction (the Company introduces itself)

2. Goals

1. Corporate Goals
2. System Goals (Product Mission)
3. Project Goals (Elevator Pitch)

3. Budget Considerations

1. Potential Benefits and Savings
2. Investment Recommendation

4. Process (Scrum)

1. Risk Management
2. Project Organization
3. Scrum Roles and Responsibilities
4. Scrum Work Flow
5. Quality Management

5. Current Situation

6. Desired State

1. Personae (Roles)
2. User-Stories
3. Workflows

7. Selection Process

1. Trial Run / Competitive Sprints

Introduction and Goals

Agile is goal and business value oriented. So the first priority of the RFP is to communicate the goals of the company, the project, the product and its users and other stakeholders. The functionality of the system is a means to the end. So the RFP started with a presentation of the company, its goals and the marketing and functional goals of the system.

This section was the subject of substantial discussion with management. Who are the intended users? How is the market changing and how should this product be positioned to react to those changes? What should the product do and what should not do? The results were realistic expectations on product and a clear vision of what should be built and why.

Budget Considerations

For agile projects, a relationship of trust between customer and supplier is essential. Trust means that each partner does not need to fear that his inadequacies will be exploited by his partners. Trust enables conflict at the level of ideas - What is best for the product, for the customer and for the customer's customers? With trust, win-win situations are possible. With trust, it is possible to give in and compromise. Establishing trust requires openness and honesty and takes time (together) to build and develop. This encouraged us to include the budget considerations directly in the RFP and presently clearly the expected risks of the project, even though it meant exposing our own warts.

It was difficult to come up with a believable model for the benefits (costs savings, new revenue, new business models etc.) of the system. There so are many consultants with spreadsheets of dubious value. So we chose to simply frame the problem. How many employees will be affected by the system? How much more business could they process if their productivity were raised by 5%, 10%, etc. Or how much money could be saved?

There was not one answer to this question, but a range of values for discussion. For each of these ranges, we calculated a value based investment recommendation, based on the double worst case analysis. [5]

Process (Scrum)

The risk analysis focused primarily on actual problems. We had done a retrospective on previous projects and had a good idea on what we wanted to improve or prevent, e.g. SW reliability problems (internal quality), fitness-for-use problems (external quality), project management and oversight problems, supplier dependencies, and misalignment of short term vs. long term budget optimizations.

Many of these issues can be addressed effectively by using Scrum, so we defined Scrum as the process for software development and for escalation management. We included a description of the Roles, Rituals and Artifacts. We included some additional metrics on Software Quality, Acceptance Tests defined and passed, Unit Tests defined and passed, which should be presented at every sprint demo.

Escalations shall be handled by a "Management Oversight Committee" modeled on Ken Schwaber's Enterprise Transition Team. Their job is to resolve impediments that the ScrumMaster can't.

Originally envisioned as two separate chapters, the Risk Analysis chapter became the introduction to the process chapter and the justification for using Scrum.

Some issues aren't directly addressed by Scrum. How do you ensure system maintenance over a 10 year period? This is both a financial question and an engineering question. We simply raised these issues with the suppliers in the RFP.

Current and Desired Situation

Next, an understanding of the context is essential to make good decisions about how to implement the product. The current situation described the process as it existed. This made clear how the business functioned and helped identify optimization potential, both in the execution process and more importantly, in the customer's decision process, which could lead to more business for the company.

The section 'Desired State' reflected most clearly the difference inherent in an agile approach. Rather than focusing on the functionality of the system, this section focused on the users (roles or personae), their goals and what they want to accomplish with the system.

We spent the most time analyzing the current and desired situations. As when planning the RFP Project, we started out by brainstorming to identify the user roles ("Personae") and their function in the process. Some of the roles were obvious: production workers for our company, our customer and our suppliers. As the purpose of this system is to reduce operating costs, these were clearly the people who would most intensively use the system.

An important discovery in the Personae workshop was that the primary (full time) users of the system are not the purchasers or influencers. The users' managers, the suppliers' and customers' sales, marketing and management and even our own top management will be 'only' occasional users, but have much more to say about the success of the system than the primary users. By giving these users real value, we create Exciter Functions [6] for those users who are most essential to success of the product.

Further, support users and the system developers also needed functionality to do their jobs, and creating the right functions (including delegating support functions back to the user) could lower operating costs substantially.

This quickly produced a draft of the desired situation, not so much in terms of the detailed functionality, but of the user roles and how they will change with the new system. It also showed how the system could change the relationships with customers and suppliers, open new business opportunities or generate new revenue from existing customers.

Specify the Product as User Stories

We ended up writing a complete set of user stories for all major users of the system. The result was 20 pages long, so we moved them into an appendix. In the main document, we just included a top level description of each user role, his/her duties and what that role expects from the system.

Augment the Description with a 4 Track Value Flow Chart

We had set out to describe the processes using Value Steam Mappings [7]. After all, we wanted to eliminate waste. However, the times involved in the processes were so variable that it was not possible to come up with numbers which would be generally accepted. Instead, we created a 4 Track Value Flow Chart. Each track corresponds to one of the primary actors:

• End Customer (Our customer's customer)
• Our Customer
• Us
• Our Supplier

Sample Three Track Value Flow Chart for a fictional job portal.

The tracks are laid one on top of the other and the horizontal axis represented time. The flow chart shows activities, processes and decisions, and who does each. The flow analysis makes it clear where productivity improvements could be made, when key decisions are made and by whom, and what interfaces between companies are present and necessary.

Selection Process

How do you know if a team can do agile? How do you know that they can work with you effectively? Which team is really the better choice? This is a hard question to answer. The classical approach suggests that the vendor should produce extensive documentation about the proposed solution (at no cost to the customer) and limit the risk through fixed price/fixed scope contracts, cost ceilings, bonus/penalty incentives etc.

These approaches do not work well with agile development projects, because they create competitive games between vendor and customer, undermine the trust which is necessary to work together effectively, and are challenged reacting to variability in scope.

Competitive Sprints, an agile selection process

Some people would like to believe that building complex software is like going to the grocery store: pick a candy bar off of the shelf, ask what it costs and decide to buy it. You get no risk and quick gratification. But building custom software is more like building a race car. A special one-off product to meet exactly the needs of its sponsor: win races.

As a customer, what are you really buying when you contract for software development? You may think you are getting a solution. But what you are really getting is an implementation team. And risk is always part of the bargain. So what you really want is a team you can trust to build your product and to minimize the risk of that choice.

Competitive sprints are a lightweight, lean approach to selecting a software development partner which should dramatically reduce the risk and cost to all parties.

Selecting a vendor through a bidding process is expensive and risky

The process of writing a huge RFP, evaluating and eventually accepting complicated bids from largely unknown vendors is wasteful, risky and expensive. A customer will often invest substantial effort into creating a Request for Proposal (RFP). I've seen RFPs whose size is measured in binders and the effort to produce them measured in man-years. A vendor will often invest comparable effort into winning a big project. On both sides, this effort produces mostly paper. These artifacts have no value except as means to the goal of selecting a vendor.

On the vendor side, this effort has to be amortized during the execution of the project itself. As there is only one winner, the others make a substantial effort and earn no money. So this risk gets passed on in the form of higher prices on successful bids.

Competitive bidding can mean offering ruinous prices. When "winning" means producing at a loss (sometimes to the point where the supplier goes out of business [8] [Automatic English translation [9]), even the customer loses. The supplier may have no choice but to play the change-request game to achieve profitability.

A Lean Approach to reduce cost and risk for all parties

How do you maximize trust and minimize risk and cost? Once trust has been established, customers are often willing to work with agile companies on a time and materials basis. Trust greatly reduces administrative overhead and enables a collaborative approach, so establishing trust should be the first priority in a project.

Lean thinking encourages us to see the whole, eliminate waste and defer decisions until enough information is available to make that decision with reasonable risk. Deferring potentially expensive decisions until their implications are clear is usually advantageous.

Simply asking for quotes and having a few talks with the sales & consulting staff is just not enough to establish trust or clarify all the open questions. Working together with the team and evaluating real results would be much more effective.

To select a vendor, after creating an agile RFP as described above:

1. Prequalify potential vendors

1. Identify potential vendors and send them the RFP.
2. Ask the vendors questions, in particular about their experience with agile methods, which will allow you to quickly eliminate uninteresting vendors.

2. Invite the survivors to bid on the project
3. Select the final short list of 2 vendors who will be invited to start work on the project.
4. Both vendors work on the project in parallel until a clear favorite is established. Both vendors get paid and both are expected to produce increments of working software according to the rules of Scrum.
5. After a few sprints, select one vendor to finish the project.

By deferring the final decision until you have actual experience with the candidates, you reduce the likelihood of picking a candidate that "looks good on paper" but cannot really deliver software.

How to prequalify vendors

Identify your candidates and send them the RFP. Ask them questions which will separate the wheat from the chaff, for example:

• Please present the team which will carry out the project. How much experience do they have with Scrum and XP (Extreme Programming)?
• Are you willing to organize the project according to Scrum (as described in the Process chapter)? What experience do you have with agile projects on this scale?
• Please estimate the user stories in story points. What is your estimate of overall complexity (size) of the system in Story Points?
• What is your expected team velocity in Story Points per Sprint?
• Given our target budget, how much of the functionality do you think can be realized?
• Given the ground rules, are you willing to participate in the competition to select the final vendor?

If the vendors are used to working on an agile basis, they will have no problems with these questions. If they are not, they will probably not even be able to respond, especially if deadlines are kept tight.

You will need to meet with prospective teams for a day or so to answer their questions about the user stories. Afterwards the vendors should be able to size the system and answer your questions quickly.

If there are more than two vendors still in the running, you will need to use the answers and the results of the interviews to trim the field down to two vendors, who then participate in the competition.

Hedge your bets on productivity differences between teams

Productivity differences between individual developers can be a factor of 10 to 20. Teams converting to Scrum often report a 3 fold increase in productivity within 6 months. The best Scrum teams have reported improvements of a factor of 10 compare to industry averages. The difference might be technical ability, but human chemistry issues are just as important, if not more so.

Even if the difference between to the top two contenders is only 25%, investing 10 to 20% of the development budget to hedge your bets reduces your risk substantially and may pay off dramatically.

Let us assume that you plan to spend $2.4 Million over 12 months, or $200'000 / Month to develop the software. If you add one month's effort to the budget, that would raise the total by 8%. But given the productivity differences between teams, even investing an additional 25% probably yields a positive ROI. Furthermore, the cost of delay while you take three months to pick a partner without producing any usable software should be much larger than the cost of redundant development for a short period.

How to run the contest

Here are the ground rules. There are two vendors. Both are going to start your project according to the process you defined in the RFP and continue for a defined trial period (probably one to three months, but not more than 25% of the total project duration). After each sprint, both players present an increment of working functionality and you will decide which partner you want to continue working with. The winner gets full compensation for the initial sprints and a contract for the rest of the project. The loser also gets paid (perhaps only 50 or 75%) and does not get a contract.

Here are the steps in the competition.

1. Agree on other playing rules: who are the team members? May staff participate who are not invoiced? Is overtime allowed? Who owns the software and ideas which are produced by the loser? You may need to ensure that quality does not get sacrificed for quantity as you will be producing code which one day will go live.
2. Agree on the definition of done. This probably should include points which confirm that the partner is capable of test driven development and continuous integration. The teams should not incur technical debt. The same definition should apply to both contestants.
3. Prioritize the backlog and, working with both vendors, create a set of fine grained stories which should be implemented during the trial period.
4. Hold the first sprint planning meeting together with both contestants, so both teams get the same initial briefing. Both teams get a defined period to implement their increments of functionality.
5. If the trial period is longer than one sprint, then continue with the Scrum process until the trial period is over. Each team should work independently.
6. Finish the / each Sprint with a Sprint Review and Retrospective. This should only include the implementation team and the product owner, as it is part of the Scrum process. When the last sprint retrospective is complete, the competition is complete.

If the teams want to give a sales demonstration, this should happen after the competitive sprints are finished.

At the end of the trial period, you have not just qualitative and quantitative data on which to base a decision. You have experience working with your new partner. You have some working software (or not). In short, you have much more information to base your decision on.

This approach does present some challenges. Managing one team can be very demanding for the product owner. Managing two teams might be difficult. This will be even more dramatic if the teams do more than one sprint, as the sprint and product backlogs will surely diverge quickly.

Who wins?

One the one hand, everybody wins.

The risks and up-front costs of producing and responding to the Agile RFP are much lower than the traditional approach. By working with the teams, you build confidence that you can work with them and that they can build the software you need. You can judge the teams based on working software rather than attractive presentations or other artifacts.

By starting to work on a solution early, you reduce your time to market. You will probably get a better solution, because the competition will spur everyone's creative juices. Even the loser will have some good ideas which will improve the final result.

On the other hand, you actually have to choose a vendor. On what criteria should you decide? Just picking the one who develops the most functionality is risky. It encourages the developers to incur technical debt to show more exciting functionality. Economic criteria will still play a role. Whatever you choose, they should ensure that you pick a vendor with whom you want to work and in whom you have confidence that they can deliver the solution you required.

Retrospective

Scrum worked well to manage the creation of the RFP. We defined the scope and met the essential objectives of the project -- even if some of the stories did not get implemented. The RFP served its purpose by focusing on the goals of its users. A definition of done was very helpful to focus on creating the document on a chapter by chapter basis. However the product owner did not really do incremental "acceptance testing", so there were a lot a changes and the end of the project.

The customer decided to keep the implementation in house, rather than contract it out, so the competitive sprint approach was not tested.

Three people worked on the RFP at approximately 50%. Approximately 4 1/2 Person-Months were invested in the RFP. Was this too much? This is the big unanswered question. In an agile process, the objective is to create the specification just in time through direct communication between team and product owner. This is not possible before a supplier has been selected.

We did learn a lot about the desired application and shifted the center of gravity of the project in ways we would not have expected had we not done the user centered design. By writing the stories ourselves, we remained business focused and not technology focused. We have a complete product backlog, but have not thought much about release strategies (time to market) because without an implementation team, we had no way to estimate the stories.

But should we really have written 20 pages of user stories? I am not sure. I had the feeling we were over-specifying. The more we produce, the more we will have to communicate to the implementation team. The more the team gets told what to do, the less they will think about the problem themselves.

Future work

This approach is a step away from a classical submission process, but is not yet purely agile. Some thoughts for next steps:

1. Repeat this process in another context. Our process had no aspirations to creating a generalized process. So your RFP might and probably look different than this project's results.
2. Use competitive sprints in an actual bidding process
3. Is it possible to involve potential vendors already in the RFP creation project? At least theoretically, it would be better to have one or more people from the implementation team involved from the beginning. Whether consultants from competing companies would have an incentive to work together is interesting question.

References (most are not working more than 10 years later...)

1. http://tinyurl.com/cnog6b
2. http://groups.yahoo.com/group/scrumdevelopment/message/30679
3. http://agilesoftwaredevelopment.com/blog/peterstev/creating-scrum-product-backlog-start-users
4. Inspired by http://blogs.techrepublic.com.com/tech-manager/?p=581&tag=nl.e053
5. http://agilesoftwaredevelopment.com/blog/peterstev/big-development-project-how-much-does-it-cost
6. http://agilesoftwaredevelopment.com/blog/peterstev/filling-product-backlog-go-excitement
7. M. & T. Poppendieck, Implementing Lean Software Development, pg. 83-92
8. http://www.vbz.ch/vbz_opencms/opencms/vbz/deutsch/DieVBZ/TramBus/VBZ-FahrzeugeImEinsatz/fahrzeuge/DasCobra-Tram/entwicklungsgeschichte.html
9. http://www.vbz.ch/vbz_opencms/opencms/vbz/deutsch/DieVBZ/TramBus/VBZ-FahrzeugeImEinsatz/fahrzeuge/DasCobra-Tram/entwicklungsgeschichte.html

Text and Images © 2009 Peter Stevens.

Based on a series of articles originally published on AgileSoftwareDevelopment.com.

---

More Agile and Project Management Knowledge

Scrum Expert

Project Management Planet

---

Click here to view the complete list of archived articles

This article was originally published in the Spring 2009 issue of Methods & Tools