FitNesse Open Source Testing Tool: A Tester's Perspective

Lisa Crispin, http://lisacrispin.com

FitNesse is an open source test framework that allows software testers, developers and customers to collaborate to create test cases on a wiki. Teams can take examples of desired software behavior and turn them into automated tests, integrated with narrative requirements documentation. FitNesse runs its own wiki web server. It's possible to test applications written in Java, .Net, Ruby, Python, C, and PHP using FitNesse.

My team has used FitNesse since 2004. This review is based on our experiences. I hope it will help you decide whether you'd like to try out FitNesse for your own test automation.

Web Site: http://www.fitnesse.org
Version tested: 20100308
License & Pricing: Open Source
Support: Yahoo group mailing list

Installation

Ease of installation is one of the strengths of FitNesse. You can download FitNesse via http://fitnesse.org. Installation is simple. For a Java installation, type java -jar fitnesse.jar. This will put all the necessary files in place. To start the server, type java -jar fitnesse.jar -p:xxxx where xxxx is your desired port number. Now you can access FitNesse via your browser with http://<hostname>:xxxx/FrontPage where hostname is your localhost or the server where you just installed FitNesse and xxxx is the port number you used. Yes, it really is that easy to get it installed.

You can write and execute FitNesse tests via the browser. Tests and suites may also be run from the command line with a RESTful syntax, or from a JUnit test.

Configuration

FitNesse comes with its own version control and test history functionality, so it can be used stand-alone. Everyone on the team can use a single FitNesse server, or each team member can import tests into their local environment, work on them, and export changes and new tests back to the main server.

You may also choose to integrate FitNesse with your existing version control, so that tests can be tagged along with the code that they test. Each team member can have their own independent FitNesse environment, and check new and updated tests into the source code control system. My team checks our tests into Subversion, and we run our FitNesse regression suites from our Hudson version control so that we get regular, speedy feedback and take advantage of Hudson's test reporting features.

FitNesse tests may be run on top of either Fit, using additional fixtures in FitLibrary, or Slim. Both currently come with FitNesse, although Fit may not be bundled with FitNesse in future versions and you'll have to download it separately. Each has its own set of test fixtures built in, with a lot of similarity between the two sets of fixtures. For example, the ScriptTable in Slim and the DoFixture in FitLibrary can both be used to create flow tests that are more readable to the business. You can use either or both, and your team will build on these built-in fixtures to automate your tests with your production code.

Documentation

The http://fitnesse.org site has an up-to-date user guide and many tutorials and screencasts to help you get started with FitNesse. There are two excellent books to help your team design effective FitNesse tests: FIT for Developing Software by Ward Cunningham and Rick Mugridge, and Test-Driven .NET Development with FitNesse by Gojko Adzic. Other websites, such as http://gojko.net and Brett Schuchert http://schuchert.wikispaces.com/FitNesse have helpful tutorials, videos and screencasts.

FitNesse has an active developer and user community, which is vital for an open source tool. If you have questions or issues, you can search the archives on the Yahoo mailing list http://tech.groups.yahoo.com/group/fitnesse/, and post your questions if you don't find an answer. You can rely on getting help from the community.

Learning Curve

Learning to write test cases in the various formats such as ColumnFixture is fairly easy. Learning how to design them well, for efficiency and ease of maintenance, takes more time. If you don't have object-oriented design skills, pair with a programmer or other team member who does. Apply good code design practices such as 'Don't Repeat Yourself' to your FitNesse tests.

If you're not a programmer, you'll also have to collaborate with a programmer who can write the necessary fixtures which take the test inputs, pass them to the production code, and send the results back to FitNesse for comparison with expected results.

Writing fixtures is fairly straightforward for programmers, as they will do this in the same language in which they write production code. For our team, it usually takes two to four hours to write a new FitNesse fixture. Occasionally, it can be tricky to come up with a fixture for a complex testing scenario.

Examples

There's a wide variety of formats for FitNesse tests, both tabular and scenario style. Our tests mainly extend the ColumnFixture. Here's a simple example of from the FitNesse user guide.

Sometimes a tabular format doesn't reflect the business example well, so there are other options. Here is an example from our own tests, using the DoFixture. The test takes out a loan with a given amount, interest rate, payment frequency, term and start date. A payment is received and processed, then the test checks the interest and principal applied and the remaining balance. This mimics the actual business flow. Slim scenario tables are similar to the Fit DoFixture.

Well-Designed Tests

FitNesse provides features that enable you to design automated tests for ease of maintenance. The !include feature is one way you can extract duplicated test code and reuse it in several tests. Variables can also be used so that when you need to make a change, you can do it in one location, and not have to change multiple test pages. Here's an example.

We can define a variable whose contents are a test table:

!define LoanSetup (!|Loan Processing Fixture|
|take loan in the amount of|${loanAmount}| with interest rate|${interestRate}|frequency|${frequency}| and term | ${term} | year with loan origination date | ${date}|
|check| periodic payment is|${periodicPaymentAmount}|
)

We can use this variable in multiple tests, substituting the desired values for loan amount, interest rate and so on for each test. For example:

!define loanAmount (1000.00)
!define interestRate (6.0)
!define frequency (Monthly)
!define term (1)
!define date (09-31-2005)
!define periodicPaymentAmount (86.07)
${LoanSetup}

When we run the test, it is as readable as the test that doesn't use variables:

The Scenario Tables in Slim provide similar functionality to this technique.

Test automation projects often fail because tests are not designed for maintainability, and the overhead to keep the tests up to date becomes overwhelming. FitNesse's design-friendly features help prevent this problem.

Add-Ons

FitNesse provides a framework for using other test tools, such as GUI drivers. This allows greater flexibility, while taking advantage of the ease of writing tests in the Wiki and FitNesse's excellent result reporting capabilities. For example, you can drive Selenium (http://seleniumhq.org/) GUI tests from FitNesse test tables. A tool called Selenesse (http://github.com/marisaseal/selenesse) provides a bridge that makes this easy for both Java and .Net environments. SWAT (http://ulti-swat.wikispaces.com/) is another example of a GUI test driver that integrates well with FitNesse. Many teams build their own custom tools using FitNesse or Slim as a base.

Drawbacks

Open source tools are only as good as their developer and user communities. With any open source tool, there may not be a reliable schedule of releases with new features. FitNesse has generally enjoyed frequent releases and improvements. My team suffered some from a backward compatibility issued in a recent release. Overall, however, useful new features arrive regularly.

We sometimes run into issues where we set up FitNesse in a new environment, without any error message or clue as to what is wrong. For example, the tests often hang with no feedback if the classpath as defined in the !path variable is invalid. We've worked around these issues by trial and error, and assume it has something to do with the architecture, but it can be frustrating.

We had a hard time integrating FitNesse into our Hudson continuous integration and build process, but the user and development community helped us find a solution. There's now a Hudson plug-in for FitNesse. You may want to make sure FitNesse will work and play well with your CI before committing to using it long-term.

FitNesse is unique (as far as I know) in using a Wiki for creating and maintaining test cases, and the Wiki can also be used as a knowledgebase and repository for story and theme requirements. The fact that anyone can contribute to the wiki is both a plus and a minus. It's a great collaboration tool, but it can easily go out of control. My team has had trouble keeping our wiki organized well enough so that we can easily find information and tests when we need it. You'll need to budget time and perhaps have a technical writer help organize and maintain the FitNesse pages and hierarchy.

It can also be tedious to create and maintain test cases on the wiki, using the wiki markup. FitNesse provides the ability to import and export test cases to and from an Excel spreadsheet, which helps. However, an IDE plugin so that tests could be edited via Eclipse or other IDEs would be a big improvement.

Benefits

Our team found early on that writing FitNesse tests forced testers, programmers and customers collaborate more. We expected it to provide a great safety net of regression tests, which it does, but we were surprised to find the greatest benefit was the increased communication during the development of a story or feature. In the process of turning business examples into FitNesse tests, we discovered disconnects in our understanding of what a story should deliver. When this happens, we can immediately get testers, developers and customers together to clarify the desired system behavior. Writing code test-first meant that we didn't have many "bugs" in the traditional sense, but the development team often missed or misunderstood requirements. The process of writing test cases in FitNesse helped us get quicker feedback and deliver the right business value.

Our FitNesse tests make superb documentation. They include both narrative about the functionality they're testing, and executable tests. They have to keep passing, so unlike the written documentation many teams have, we're forced to keep them up to date. When someone from customer support comes over to ask what results should come out from a given set of inputs in production, we don't get into a philosophical discussion over how the code works, or scratch our heads trying to remember. We can prove how the code works with a passing FitNesse test. Maybe this isn't how the business really wants the code to work, and in that case, they can write a story to change it. This saves lots of time, and makes us look smart.

The 'Drawbacks' section in this review is longer than the 'Benefits' section, but for our team, the benefits far outweigh the drawbacks. As I wrote this article, one of our FitNesse suites caught a regression bug 20 minutes after a code change was checked in. We'd have found the regression manually at some point, but it may have taken up to 24 hours. The fast feedback helps us maintain a sustainable pace and ensure we can deliver high quality code to production anytime.

---

More Software Testing resources

• Software Testing Magazine
• Functional Testing Tools Directory
• Software Testing Tutorials and Videos

---

Click here to view the complete list of tools reviews

This article was originally published in the Summer 2010 issue of Methods & Tools