Click here to view the complete list of archived articles
This article was originally published in
the
Spring 2005 issue of Methods & Tools
Click
here to reach PDF area
Improving Application Quality Using Test-Driven Development (TDD)
What is the one activity or phase that improves the quality of your application? The answer is an easy on: Testing, and plenty of it.
But when do we traditionally perform testing? If you are following a "waterfall" style approach to software development, it’s very likely that you have a testing phase somewhere towards the expected end of the project. This is peculiar, because we all know that at this stage in a software development project, the cost of any code or requirement changes is known to be much higher. Figure 1 presents us with the five major phases of the waterfall approach. I have superimposed Barry Boehm’s "Cost of Change"[1] curve onto Figure 1.
Figure 1: Traditional "Cost of Change" curve with the waterfall model superimposed
Clearly, if we can move the testing phase further to the left we might stand a chance of reducing costs whilst enjoying the natural effect of a longer testing period. TDD helps us achieve this by embracing agile techniques that sport iterative or incremental development. Instead of performing the five waterfall stages noted in Figure 1 (typically horizontally, left to right, i.e. requirements gathering then analysis & design, etc), we perform the stages vertically, bottom up, i.e. perform a little of each stage until we are complete.
By embracing iterative development, TDD allows us to be much more flexible towards our client’s ever-changing requirements. Instead of being in the position where we had spent a lot of waterfall time developing a product or feature, where the cost of change is very high, we find ourselves in the enviable position whereby we can demonstrate parts of the product or feature much earlier. By demonstrating features earlier, the cost of the inevitable changes is much lower. Of course, knowing about requirement changes earlier in the process mean that important architectural decisions (i.e. those that make requirement changes much harder to implement) have yet to be made or are at least in their infancy and are not the brick wall they might be.
Over the course of this article I will demonstrate how TDD can improve your application’s quality by introducing testing much earlier into the process. Additionally, I hope that the agile world’s tenet of flexibility and acceptance that client’s will change their mind during the course of a project, will become clear. By maintaining a suite of tests that are both repeatable and automated, we can enjoy the luxury of being able to make sweeping changes to our code with the knowledge that any deviations and bugs will be picked up by re-running the test suite.
What is TDD?
TDD is radical process that promotes the notion of writing test cases that then dictate or drive the further development of a class or piece of code. This is often referred to as "writing tests first". Indeed, I will refer to the phrase "writing tests first" as one of TDD’s primary principles.
Whilst I refer to TDD as a "process", followers of Agile communities including XPers (eXtreme Programming), will know TDD as an integral "core" practice. Appreciating that many of the XP-oriented books push TDD as a primary part of their content, we are lucky enough to be able to adopt TDD as a process that can be "bolted" on to our existing software development process, regardless of whether it is Agile, waterfall, or a mix of the two. There may well be entrenched differences in the semantics of the words practice and process. I believe that TDD in an Agile/XP environment, perhaps were Scrum is practiced, TDD can also be practiced. However, outside of a strictly Agile environment, TDD is a sub-process that becomes part of the development process.
TDD is radical because traditionally developers either test their own code, or they have a testing group that undertakes this task. Sadly developers who test their own code are to close to the problem and the solution; they have a personal ego investment that has to be looked after. Thus developers rarely perform as much testing as we would like.
In fact, the first time a developer tests his/her code, it’s likely to be heavily tested. However, the second time a developer is asked/expected to test the same piece of code/functionality, testing will not be as involved. By the time they are asked to test a third and fourth time, boredom sets in and the quality and coverage of testing reduces considerable. As a developer myself, I know this has happened to me and an informal poll of my colleagues in the community revealed similar results.
If you are building a product using the C# language, typically TDD suggests that you’ll write and maintain a set of tests also written in C#. Ideally the tests will exercise all of your production code. One of the TDD catch phrases is: "no code goes into production without associated tests". That is a fairly tall order, however if you are able to achieve it you will be able realise high quality code, with any bugs being caught before they are shipped.
If you have been reading Martin Fowler’s works about refactoring, TDD lends itself to the refactoring process. After all, refactoring is about changing a class/method’s internal design without affecting the external behaviour. What better way to ensure your refactoring hasn’t broken anything than with a solid, reusable set of test cases? I know that I’m not alone – we’ve all written some code, tested it, then re-written it (or "improved" it!) only to find that the re-write introduces problems elsewhere in the application.
TDD is a different mind-set. It is the ‘tests’ that ‘drive’ the ‘development’, not the other way around. By writing a test case for a scenario, your test either dictates your implementation or gives you some good pointers. It might not be the ideal implementation, but that’s part of the TDD process – small, fine-grained steps lead to increased developer confidence which results in larger steps downstream.
Removing The Boredom Of Testing
Notice that I used the phrase: "writing tests first". If we are to remove the boredom of the testing process, we need to codify the tests that we would perform. By writing test cases, typically in the programming language that the application or class under test is written using, we are essentially automating the test process.
Automating the process is the key. If we can make the tests easy to perform once, twice, three and four times, ideally at the click of a button, developers will be happy to run the tests frequently. Of course, each time we run the tests, we are performing the same tests, over and over again, the tests are said to be repeatable.
Repeatable testing is also a major benefit. Were a developer performs manual testing, we know from experience that they test less each time. Similarly, when asked to perform a particular test, unless there is a very precise test script available, the chances are that a human-driven non-trivial test will be performed identically each and every time.
TDD Frameworks and Tools
TDD has origins in the Java and Smalltalk camps. Originally there was a tool by the name of SUnit (for Smalltalk). That was followed closely by a tool called JUnit (for Java). Since then, the term ‘xUnit framework’ has been used to represent language agnostic versions of the same tool. The xUnit framework has been ported to many different platforms and languages.
NUnit is a .NET implementation of an xUnit tool or framework that works with C# and Visual Basic.NET amongst others. Outside of the Microsoft IDEs, Borland’s Delphi has its own xUnit testing framework called DUnit. Given the uptake of Microsoft .NET, over the course of this article I will demonstrate NUnit using a C# example.
Prerequisites
I will assume that you have downloaded and installed NUnit (available from http://www.nunit.org) and that you have a C# IDE installed (e.g. Microsoft Visual Studio 2003, Delphi 2005 or C#Builder). I am using Microsoft Visual Studio 2003, if you are using a different environment the screenshots will vary.
Getting Started
I will assume that you have created a new, blank, C# Windows Application, as shown in Figure 2.
Figure 2: Creating a new Windows Application using Visual Studio 2003
I mentioned earlier that TDD expects us to write test cases before we write any code. Therefore, instead of attempting to dive into the detail of developing a full-blown application, we will concentrate on developing a test case that will dictate how we might implement a small piece of functionality. This leads us to another TDD principle, borrowed from the eXtreme Programming world: do the simplest thing that could possibly work.
Writing a test case necessitates the use of the NUnit framework, so we must add a reference (using the menu Project, Add Reference) to the nunit.framework assembly as shown in Figure 3.
Figure 3: Adding a reference to the nunit.framework assembly
If you recall the first TDD principle that I mentioned, write tests first, that is our next step: we will create a new class that will act as a holder for one or more tests. The simple application that we will develop over the remainder of this article will be a simple magazine indexer. A magazine index comprises of zero or more magazines, a magazine comprises of zero or more articles.
In a production application, the architecture might end up being somewhat complex, possibly involving web access, multiple tiers, expensive databases and a lengthy development timeframe. Our example will be less complex and will be demonstrable in a very short space of time, albeit it will be bound by the fact that we are doing the simplest thing that could work.
So, the simplest thing that could work is a test case that assumes code has been written to handle our magazine indexing needs. Listing 1 presents us with the code required to create a new magazine index, add a magazine, and then check that the magazine index was created correctly. Of course, we have not written the class MagazineIndex yet, so you would be right to think that the compiler is going to complain!
Indeed, this is another TDD principle: let the compiler tell you when something is wrong. Compiler time is cheap; we can afford to let the compiler find our mistakes for us. For the sake of this demonstration, I have not worried about how an instance of MagazineIndex is actually created (there is a reason for this as we will see shortly).
namespace WinApp
{
Using System;
using NUnit.Framework;
[TestFixture]
public class TestMagazine
{
[Test]
public void CheckNew()
{MagazineIndex cbtMagazineIndex;
cbtMagazineIndex = null;
Assert.IsNotNull(cbtMagazineIndex, "cbtMagazine is null!");}
}
}
Listing 1: Writing Tests First, doing the simplest thing that could possibly work
Anatomy of Listing 1
Listing 1 demonstrates a number of the NUnit framework features: I have highlighted these in bold.
Firstly, we must add a reference to the NUnit Framework assembly: this gives us access to a number of attributes, such as [TestFixture] and [Test] as well as the Assert class. The Assert class is rather powerful and forms the basis of NUnit testing framework.
Secondly, we must use the NUnit attribute [TestFixture] to indicate that the following class, in this case TestMagazine, will contain test cases that NUnit should be aware of (there will be more about precisely how NUnit achieves this later in this article).
Thirdly, we must provide a method, the name of which does not matter, that is prefixed with the [Test] attribute. Methods such as these should perform a small test that uses the NUnit Assert class to indicate success or failure. The Assert class provides us with a number of methods that can be used to determine whether a test passes or fails. These are: AreEqual, AreSame, Fail, Ignore, IsFalse, IsNotNull, IsNull, IsTrue and ReferenceEquals. As you can see from Listing 1, I have used the IsNotNull method to confirm that the creation of cbtMagazineIndex took place without failure.
However, it is important to note that listing 1 does not actually compile. The first line of the CheckNew() method attempts to declare cbtMagazineIndex (cbt standing for class being tested) as an instance of the class MagazineIndex. Clearly, we have not mentioned MagazineIndex yet, hence the compiler will complain. Luckily, using the compiler helps us and it is one of TDD’s primary facets.
Use the compiler…
By examining the CheckNew() test from Listing 1, we can see that this [Test] requires a MagazineIndex class. What is the simplest thing we could do to get this line though the compiler? Well, we could simply declare an empty class, which would get us to the point where Listing 1 will compile.
namespace WinApp
{
public class MagazineIndex
{
}
}
Listing 2: Empty classes can solve problems
Now that Listing 1 compiles, we know it does not do very much, what is next? Listing 1 contains a single NUnit [Test], we can use .NET’s reflection capabilities to examine assemblies (.exe or .dll) for [TestFixture] classes and [Test] methods within those classes. Indeed, NUnit is supplied with a tool that does just this; it is called NUnit-GUI.
Test Visualisation
TDD aficionados use the phrase "red/green/refactor" when practicing TDD. The colours relate to the success or failure of tests – a test that fails is considered red, and a test that passes is green. To help us manage our tests, most xUnit frameworks are supplied with a ‘test runner’ application. NUnit is no exception – if you look in C:\Program Files\NUnit 2.2\bin\ you should find nunit-gui.exe. NUnit GUI is a graphical front-end that takes a standard .NET PE format assembly, interrogates it, then allows us to run any tests contained in the assembly.
I’ll assume that you have double clicked on nunit-gui.exe and that it is running. Using Explorer (or your favourite file system navigation tool), locate the .exe that Visual Studio 2003 created (mine is in the Example\bin\Debug directory), then drag the .exe onto the NUnit-GUI window.
There is a ‘Run’ button – click on this button. Figure 4 should look familiar – this is what is known as the infamous ‘red bar’ whose partner and opposite in TDD world is the ‘green bar’.
Figure 4: NUnit’s GUI front-end
Taking a closer look at Figure 4 and we can see that the NUnit-GUI application has interrogated our WinApp.exe extracting the [TestFixture] TestMagazine and the [Test] CheckNew on the way. When the NUnit-GUI application then ran the tests, it was able to provide us with a list of Errors and Failures on the tab of the same name. Better than that, the failure message that we included in the test appears here to.
Getting the test to pass is the simple matter of creating an instance of MagazineIndex as shown in Listing 3.
[Test]
public void CheckNew()
{
MagazineIndex cbtMagazineIndex;
cbtMagazineIndex = new MagazineIndex();
Assert.IsNotNull(cbtMagazineIndex,
"cbtMagazine is null!");
}
Listing 3: Making the test pass
With Listing 3 in place, re-compiling and then re-running the tests through NUnit-GUI reveals Figure 5, the green bar.
Figure 5: Green bar, test OK
As we will see shortly, another TDD principle revolves around the fact that a test should fail the first time it is executed. Unfortunately, because of the blatant simplicity of this first example, the CheckNew() test actually passes!
Adding Functionality
What we’ve seen so far, whilst sound TDD practices, hasn’t resulted in much demonstrable code. We know that MagazineIndex is going to manage a collection of zero or more instances of the class Magazine, so it stands to reason that we might have an AddMagazine method. So let us set about writing the test for AddMagazine.
Listing 4 presents our new test case – remember, we must write our tests before we write any implementation code! I’ve done the simplest thing possible, added CheckAddMagazine that creates an instance of MagazineIndex, adds a magazine name, and then checks that the magazine name was added correctly. Listing 4 is not going to compile – there’s no AddMagazine method defined, nor is there an IsMember function.
[Test]
public void CheckAddMagazine()
{
MagazineIndex cbtMagazineIndex;
cbtMagazineIndex = new MagazineIndex();
cbtMagazineIndex.AddMagazine("Methods & Tools");
Assert.IsTrue(cbtMagazineIndex.IsMember("Methods & Tools"), "Magazine was not added to collection!");
}
Listing 4: Adding functionality
Doing the simplest thing to Listing 2 will allow our new test in Listing 4 to compile. This means adding the AddMagazine and IsMember methods to the class MagazineIndex. Listing 5 presents MagazineIndex with these modifications.
namespace WinApp
{
public class MagazineIndex
{public void AddMagazine(string MagazineName)
{
}public bool IsMember(string MagazineName)
{return false;
}
}
}
Listing 5: Doing the simplest thing
With Listing 5 in place, we can re-compile the solution and re-run the test. Prepare yourself for some bad news.
Figure 6: Red bar, test fails
Bad news, the test failed. What do we need to do in order to fix this? What might move us in the right direction would be a change to the AddMagazine and the IsMember methods, and the introduction of a string called MagazineList, this is the simplest thing. Listing 6 presents the updated MagazineIndex class.
public class MagazineIndex
{
private string MagazineList;
public void AddMagazine(string MagazineName)
{MagazineList = MagazineName;
}
public bool IsMember(string MagazineName)
{return MagazineList == MagazineName;
}
}
Listing 6: Small steps, tests pass
Figure 7: One magazine, the test passes
Adding Two Magazines
Like all good developers, we will write some test code that exercises adding more than one magazine. Listing 7 presents a new test that does just this.
[Test]
public void CheckAddTwoMagazines()
{
MagazineIndex cbtMagazineIndex;
cbtMagazineIndex = new MagazineIndex();
cbtMagazineIndex.AddMagazine("Methods & Tools");
cbtMagazineIndex.AddMagazine("The Delphi Magazine");Assert.IsTrue(cbtMagazineIndex.IsMember("Methods & Tools"), "M&T was not added to collection!");
Assert.IsTrue(cbtMagazineIndex.IsMember("The Delphi Magazine"), "TDM was not added to collection!");
}
Listing 7: Adding two magazines to the collection
Figure 8: Two magazines, the test fails
Our simple approach of using a string to manage the MagazineList has caused a test to fail. That is part of the TDD process – let the tests dictate (drive) which code is developed and how it is developed. In this case, CheckAddTwoMagazines dictates that we need to develop a collection or list of some sort to replace the simple string. The TDD mantra of "red, green, refactor" seems to stand true. It is time to refactor the AddMagazine and IsMember methods.
namespace WinApp
{
using System.Collections;
public class MagazineIndex
{private ArrayList MagazineList = new ArrayList();
public void AddMagazine(string MagazineName)
{MagazineList.Add(MagazineName);
}
public bool IsMember(string MagazineName)
{return (MagazineList.Contains(MagazineName));
}
}
}
Listing 8: Introducing an ArrayList
After the introduction of Listing 8 , re-compiling and re-running the tests reveals Figure 9 and a green bar.
Figure 9: Two magazines and a green bar
ArrayLists in .NET languages are a useful mechanism that allows us to manage groups of object references, in our case strings. An ArrayList has the ability to grow as new object references are added – hence it is the ideal abstraction for managing a collection of magazine titles. ArrayLists support useful methods like Add, Insert, Remove, and conveniently for us a Contains method (we have used this in our IsMember implementation.)
Smells
Now that we have three tests in place, a
certain amount of duplicated code has appeared. Most obviously, there is the
class setup code:
MagazineIndex cbtMagazineIndex;
cbtMagazineIndex = new MagazineIndex();
We know that refactoring plays a major part in the TDD process, but so far we’ve only really looked at refactoring the implementation class. Duplicated code is something XPers call a ‘bad smell’, and it is something that should be refactored out. Conveniently, most xUnit frameworks implement a "setup" and "teardown" mechanism. NUnit uses the attributes [SetUp] and [TearDown].
The [SetUp] attribute allows us to specify a method that will be called once at the creation of the [TestFixture]. We can use it to create an instance of the MagazineIndex class.
The [TearDown] attribute allows us to specify a method that will be called when the [TestFixture] is destroyed. We could use this to clean up any objects/memory that are not garbage collected.
Think of [SetUp] and [TearDown] as "initialise" and "cleanup", or "create" and "destroy/free".
If we refactor our most recent test class, we can remove the duplication by making the class being tested a private member of the class TestMagazineIndex. The introduction of a SetUp method then allows us to construct a new instance of MagazineIndex.
namespace WinApp
{
using System;
using NUnit.Framework;
[TestFixture]
public class TestMagazine
{private MagazineIndex cbtMagazineIndex;
[SetUp]
public void Setup()
{cbtMagazineIndex = new MagazineIndex();
}
[TearDown]
public void TearDown()
{cbtMagazineIndex = null;
}
[Test]
public void CheckNew()
{cbtMagazineIndex = new MagazineIndex();
Assert.IsNotNull(cbtMagazineIndex, "cbtMagazine is null!");}
[Test]
public void CheckAddMagazine()
{cbtMagazineIndex = new MagazineIndex();
cbtMagazineIndex.AddMagazine("Methods & Tools");
Assert.IsTrue(cbtMagazineIndex.IsMember("Methods & Tools"), "Magazine was not added to collection!");}
[Test]
public void CheckAddTwoMagazines()
{cbtMagazineIndex = new MagazineIndex();
cbtMagazineIndex.AddMagazine("Methods & Tools");
cbtMagazineIndex.AddMagazine("The Delphi Magazine");Assert.IsTrue(cbtMagazineIndex.IsMember("Methods & Tools"), "M&T was not added to collection!");
Assert.IsTrue(cbtMagazineIndex.IsMember("The Delphi Magazine"), "TDM was not added to collection!");}
}
}
Listing 9: Refactoring followed by re-running of tests
Of course, with such a sweeping change like that of Listing 9, we need to run the tests again to be sure that everything still works. Luckily the tests pass and we get a green bar.
One Assertion per Test?
The TDD Yahoo group [2] contains a long running thread about the merits of how many assert statements a test may have. Whilst the hardened core of TDDers will argue that "one test should test one aspect", you may have noticed that I have included two assert statements in one of my tests. I believe with practice I’ll reach the point whereby I have a single assertion per test, but until I’ve taken TDD across an entire [large] project, I’ll have to continue using one or two assertions per test. So far I’ve enjoyed considerable success introducing TDD into the new classes that I’m writing for inclusion in existing applications. I’m also taking the time to write tests for some of the periphery classes as I go.
The TDD Yahoo group is the place to find Dave Astels and many other TDD gurus – if you’re serious about TDD it’s definitely a group to follow.
Naming Conventions
Many xUnit implementations enforce a naming convention. Tests, for example, must be prefixed with the word ‘Test’, e.g. public void TestCreateNew(). .NET’s attributes relieve us of that convention; we are able to essentially prefix classes and methods with the [TestFixture] and [Test] attributes. These attributes are embedded in the assembly (our .exe) which can then be interrogated by the NUnit GUI / Test Runner application.
Equally, we could examine the assembly using other tools. Lutz Roeder’s .NET Reflector [3] will let you examine the ‘metadata’ inside a .NET assembly.
However, there is one small NUnit convention that we must follow. All NUnit [Test] methods must be declared as public void. If you attach the [Test] attribute to a non-public method, NUnit will simply ignore it.
Conclusions
As you may have gathered, refactoring and TDD are made for each other. During the refactoring process we find ourselves making sweeping changes to the internals of a system, we need a mechanism that ensures that any refactorings do not break the functionality, TDD provides us with that mechanism.
I noted a number of principles:
These principles are worth following. Granted, over the course of this article, I have deliberately taken these principles to their extreme (small steps and do the simplest thing), your initial work with TDD should take similar small steps. As your confidence increases, you will find that larger steps are possible. However, taking steps that are too large will take you back to square one: biting off more than you can chew is why we spend a lot of time scrapping code and re-working it.
I have used a very simple example in this article. If you would like to see a more detailed example please visit my web-site[4] where you can find examples written using Visual Basic.NET and Delphi.
I hope this article has given you a flavour of the power of TDD – the ability to change a class’s implementation and re-test it with just a few mouse clicks is a great confidence boost. All too often traditional non-automated testing techniques involve the original developer re-writing some code, re-testing the bit that changed with little regard for the periphery code that really should be tested too. Maintaining an exhaustive set of tests that can be run again and again means those obscure bugs that usually slip through the net are caught whilst the original developer is focusing on the problem – not hours, days or sometimes weeks after the code was re-written/tweaked.
If you like the idea behind TDD but do not wish to use Visual Studio .net, I can recommend Dave Astels site[7] and Ron Jefferies site[8] as good places to start looking for a TDD framework for your environment.
Lastly, I will leave you with the thoughts of Rutherford[5] and Marick[6]. Testing is an emotive subject, whilst this article has used the phrase test-driven development, I agree with Rutherford and Marick, the code that makes up a [TestFixture] and a [Test] provides us with an example of how a class/method can be invoked and what we might expect it to return. Given that, it is fair to say that [TestFixture]s and [Test]s are the best live documentation you will have for your code-base: think about it, paper-based documentation is rarely in sync with the code-base, TDD’s repeatability ensures the [Test]s are.
Resources:
[1] http://c2.com/cgi/wiki?ExponentialCostCurve
[2] http://groups.yahoo.com/group/testdrivendevelopment
[3] http://www.aisto.com/roeder/dotnet
[4] http://www.craigmurphy.com
[5] http://cgi.bramwell.plus.com/krblog/2005/02/test_driven_is.html
[6] http://www.testing.com/cgi-bin/blog/2003/08/21#agile-testing-project-1
[7] http://www.adaptionsoft.com/tddapg_xunit.html
[8] http://xprogramming.com/software.htm
Slide-deck accompanying this article: http://www.craigmurphy.com/bug/tdd/13Jul2004/TDDIntro.zip
Examining the Cost of Change, Scott Ambler: http://www.agilemodeling.com/essays/costOfChange.htm
Books:
Test-Driven Development: By Example, Kent Beck,
Addison-Wesley, 2003, ISBN 0-321-14653-0
test-driven development: A Practical Guide, Dave Astels,
Prentice-Hall/Pearson Education, 2003, ISBN 0-13-101649-0
(reviewed here: http://www.craigmurphy.com/bug/tdd/review.htm)
Refactoring: Improving the Design of Existing Code, Martin
Fowler,
Addison-Wesley, 1999, ISBN 0-201-48567-2