TestNG - an Open Source Java Testing Framework

Tomek Kaczanowski, http://practicalunittesting.com

TestNG is general purpose open source Java testing framework, not limited to unit tests. As a natural competitor to JUnit, it trumps its predecessor with features better suited for integration and end-to-end testing, still being as strong as JUnit in the field of unit tests.

TestNG supports parameterized tests out-of-the-box (in much more convenient way than JUnit does), it facilitates running multi-threaded tests, and allows to express dependencies between test methods.

Web Site: http://testng.org
Version Tested: TestNG 6.4 on Linux with Java 1.6.0_26
License & Pricing: Open Source, Apache 2.0
Support: User mailing list http://groups.google.com/group/testng-users

Using TestNG

TestNG is distributed as a single JAR file, which must be available in the classpath in order to execute tests. It integrates very well with the most popular build tools: Ant, Maven (via Maven Surefire Plugin) and Gradle. TestNG is also supported by all major IDEs (in some cases the installation of additional plugin is required), can be used with different JVM languages (e.g. Java, Groovy, Scala) and cooperates with many quality and testing tools (e.g. code coverage tools, mocking libraries, matchers libraries). Some popular solutions - e.g. Spring Framework - provide means to facilitate testing with TestNG.

This introductory article concentrates on the TestNG features. Please consult TestNG documentation for installation instructions.

Developers Testing

The idea that developers have to test their own code has gained ground in recent years. Numerous examples of complex systems tested almost solely by developers prove the usefulness of developers tests. Unit tests help to maintain a sound codebase, which is crucial for its maintenance and further development. Integration and end-to-end tests allow to verify if the produced software meets client's requirements. TestNG can be used to implement all kinds of tests, thus improving both the internal and the external quality of the implemented systems.

Let he Code Speak

We start our journey into the world of TestNG with a very simple example. We have a class which represents money (named - surprise, surprise! - Money), and we want to test whether its add() method is properly implemented. A simple TestNG test could look as follows:

import org.testng.annotations.Test; [1]
import static org.testng.Assert.assertEquals; [2]

public class MoneyTest {

	@Test
	public void shouldAddSameCurrencies() {
    		Money moneyA = new Money(4, "USD"); [3]
    		Money moneyB = new Money(3, "USD"); [3]
    		Money result = moneyA.add(moneyB); [4]
    		assertEquals(result.getAmount(), 7); [5]
	}
}

Two imports are required:

[1] one for @Test annotation which marks test methods (that is, methods which should be executed by TestNG),

[2] assertion methods, which are later used to verify our expectations.

The test method itself is very simple: it constructs the required objects [3], invokes the method in question [4], and verifies its result [5].

If the assertion is not met then TestNG provides a detailed information on the cause, for example:

java.lang.AssertionError:
    Expected :7
    Actual   :6

TestNG also provides a stacktrace, which is not printed here for the sake of brevity. The stacktrace makes it clear which line of test caused the test to fail. This allows developer to quickly navigate to the failed test, and start fixing things.

Based on this example we can see that TestNG works similarly to other testing frameworks. Now let us take a look at some more advanced and interesting features of TestNG.

Parameterized Tests

Unlike other testing frameworks, TestNG allows to pass parameters to test methods. . The following snippet of code illustrates this idea:

@DataProvider [1]
private static final Object[][] getMoney(){
    return new Object[][] {
        {new Money(4, "USD"), new Money(3, "USD"), 7},
        {new Money(1, "EUR"), new Money(4, "EUR"), 5},
        {new Money(1234, "CHF"), new Money(234, "CHF"), 1468}};
}
    
@Test(dataProvider = "getMoney") [2]
public void shouldAddSameCurrencies(Money a, Money b, 
        int expectedResult) {
    Money result = a.add(b);
     assertEquals(result.getAmount(), expectedResult);
}

The purpose of the first method - getMoney() - is to provide data for the actual test method - shouldAddSameCurrencies(). - When we run this test, we execute three tests, each with different set of parameters. The first test would be run with US Dollars, the second with Euros, and the third with Swiss Francs. This is achieved with two simple annotations: [1] the dataProvider attribute of the @Test annotation informs TestNG that it should ask the [2] getMoney() method to provide data. Such construct allows to execute many test cases at once without introducing repetition of code.

Data providers nicely decouple the "business logic of test" from the test data. The logic to be tested is enclosed within the test method, while the data is kept separately.

Test Fixture Setting

The setting of the test context (so-called test-fixture) is one of the challenges of tests writing. TestNG helps by providing many methods which allow to control when the objects are being created. This is very important especially for integration tests, which often use resources (e.g. web servers or database connections) that take time to create. An example is presented below. A server is created before the whole test suite is executed [1], while a client object is created before every test method [2]. This way a costly resource is set up only once, and client (a class we test) is in a clean state before any of its method is executed.

Additionally, the timeOut attribute of the @Test [1] annotation will cause the setUp() method to fail if it does not finish before timeOut (which is set to one second in this case). This is another useful feature of TestNG which allows to avoid tests hanging forever.

@Test
public class TestFixtureTest {
    
	private Server server;
	private Client client;
    
	@BeforeSuite(timeOut = 1000) [1]
	public void setUpServer() {
    	    server = new Server();
    	    server.start();
	}
    
	@BeforeMethod [2]
	public void setUpClient() {
    	    client = new Client();
    	}
    
	// some test methods here
    
	@AfterSuite
	public void shutDownServer() {
    	    server.stop();
	}
}

Custom Listeners and Reporters

In case of all kind of tests, it is very important that developers get a very precise information on the results of test execution. TestNG not only provides a decent report by default, but also facilitates to tweak it. It is up to developers how they use this feature: printing some additional information during the execution of tests (e.g. time of execution of the last test method), taking screenshots after each failed Selenium test, or writing test results into an Excel spreadsheet.

Parallel Tests Execution

More and more of the code we write should work when accessed by multiple threads at once. TestNG provides some support for testing this type of code as we can see below. Using the threadPoolSize and the invocationCount attributes of the @Test annotation [1] we instruct TestNG to execute the same test method 100 times by 7 threads simultaneously. This way we can find out if our IdGenerator really generates unique IDs (at least within a single JVM). This test method relies on the specific set implementation provided by the HashSet class, which returns true only if an unique object was actually added to it [2]. The parallel execution of this test method by many threads is done with the use of a single annotation and does not require any additional coding.

public class IdGeneratorTest {

	private IdGenerator idGen = new IdGenerator();
	private Set ids = new HashSet(100);

	@Test(threadPoolSize = 7, invocationCount = 100) [1]
	public void idsAreUnique() {
    		assertTrue(ids.add(idGen.nextId())); [2]
	}
}

Groups and Dependencies Between Tests

Grouping tests is another very useful feature of TestNG. Adding appropriate annotations to test classes or test methods, you could easily divide your tests into "fast" and "slow" groups), and execute them independently.

While it is usually advisable to keep unit tests completely independent from each other, this is not the case for integration and end-to-end test. TestNG recognizes this difference, and provides convenient ways to implement such scenarios. It is possible to express dependencies between groups of tests and/or individual tests. The dependent test methods would be run only if all test methods, they depend on, succeeded. This could be used to implement many interesting scenarios., You could run integration tests only if unit tests passed or exhaustive system tests only if smoke tests succeeded. This helps to shorten the feedback loop, which in general is a good thing.

Conclusions

The old saying says that you should "use the right tool for the job". In these days of strong focus on automated tests, and teams moving towards Continuous Deployment, it is even more important that the developers use the best tools available. I hope to convince you that TestNG is indeed the right tool with this article that presented only a subset of TestNG features.

Let us finish this short overview with a summary of TestNG capabilities. TestNG:

• supports all kind of developer tests: unit, integration and end-to-end tests,
• is robust and frequently released
• is well documented, and has a vivid users community,
• has a strong development leader - Cedric Beust - who makes sure the development of TestNG does not deviate from the right path,
• trumps JUnit in many aspects, especially when it comes to integration and end-to-end tests,
• has a flat learning curve,
• is well supported by the vast majority of tools.

Further Readings

TestNG website: https://testng.org

TestNG official documentation: https://testng.org/doc/documentation-main.html

TestNG users mailing list: https://groups.google.com/group/testng-users

Next Generation Java Testing: TestNG and Advanced Concepts, book by Cedric Beust https://testng.org/doc/book.html

Practical Unit Testing with TestNG and Mockito, book by Tomek Kaczanowski, http://practicalunittesting.com

---

Unit Testing Articles

Writing Testable Code

More Software Testing and Java Knowledge

Unit Testing Tools

Java Tutorials and Videos

Software Testing Magazine

---

Click here to view the complete list of tools reviews

This article was originally published in the Spring 2012 issue of Methods & Tools