Timur Strekalov, Zimory, http://zimory.com/
Web Site: http://timurstrekalov.github.com/sagaVersion tested: 1.3.3
Java 6+ for the core and the CLI tool
Maven 2.2.1 or 3.0.3 for the Maven plugin
Gradle 1.2 for the Gradle plugin
Any modern browser (Google Chrome, Mozilla Firefox, Safari) for viewing the HTML reports
License & Pricing: Open Source, Apache License, Version 2.0, Free
Support: Issue tracker at https://github.com/timurstrekalov/saga
Writing automated tests is an essential part of software development, and unit testing the simplest and fastest way to prove the validity of your code. Since unit tests are designed to verify the correctness of specific units of your application in separation (such as classes in object-oriented programming), it is rather beneficial to know how much of the given unit is actually covered by the test.
Some teams establish certain standards for their code, setting a percentage of the code that must be covered by unit tests (e.g. 90%), and code coverage tools such as Saga allow them to track the quality of their code. Moreover, some teams configure their Continuous Integration servers, such as Jenkins, to fail the build if it does not meet the acceptable code coverage threshold.
Saga was envisioned and designed from ground up as a tool that would allow people of very different skill sets, not only software developers, to customize their build cycle and use the appropriate implementation to suit their needs, without too much effort. It is highly configurable, but strives to provide sensible defaults for most of the parameters, requiring only three configuration items to be specified in order to produce a coverage report.
- point Saga to the base directory of their source codebase, such as "src";
- provide at least one name pattern of the tests to be run, such as "**/*Test.html;
- specify the output directory for the coverage report, such as "target/coverage"
That's it. Three very basic things one would want to configure anyway, and it is enough to free the user of the worries of the details such as instrumentation and let them have what they want.
Two types of coverage reports
Saga can generate two different types of reports:
- Total coverage report – a coverage report indicating all statements that were covered by all the tests that had been run.
- Per-test report. Unit tests are designed to cover a specific unit of your code, and it is very desirable that the coverage of one unit would not be affected by another unit's tests. While a total coverage report can show that a unit has 100% coverage, it might mean that it was referenced by tests that are not really testing this particular unit – it might even be possible when there are no tests for that unit.
A per-test report shows only the coverage of the code by each unit test. This is why it might be useful for determining whether the test that was designed to validate a unit is actually properly covering it.
This is exactly why Saga allows you to configure the code that you would like to have covered in order to discover the units that remain hidden from all your unit tests, allowing for the most accurate code coverage results.
There is an idea to allow running testing scripts directly, but this feature is currently not supported in version 1.3.3. Saga can only execute tests that you would be able to execute using a normal web browser.
Saga is by default configured to execute tests concurrently, making use of all of the computing power available in the machine. Instrumentation itself takes a fraction of the time, as does the fact that instrumented code gets somewhat slower due to the extra statements that are the core of code coverage generation being introduced into the code; all of this adds up, so making use of all of the cores of the CPU makes the whole process much faster.
Neat coverage report
By default, Saga produces two different types of coverage reports: one in HTML and one in a raw LCOV format, which can be read by Sonar, for example. The former, however, is a very slick interactive report that shows various code coverage statistics and allows you to see the actual statements that were covered and missed. It was designed by Marat Dyatko, an engineer with a great taste (when it comes to design), to whom I owe a debt of gratitude for taking the time to make the HTML report look as great as this:
Starting with version 1.3.3, Saga also provides support for generating PDF and CSV reports in addition to the HTML and LCOV formats.
Saga integrates with build tools through a plugin system, so you should be able to get all the features with any implementation that fits your needs. Saga currently has implementations for two major build tools, Maven and Gradle.
If you are using any build tool other than Maven or Gradle it might seem like you cannot use Saga, which, luckily, is wrong. First of all, you should consider submitting a feature request by going to the GitHub page and creating a new issue, or better yet, if you have experience with the concerned build tool and have the time to contribute an implementation, I will be happy to help you with it. In the meantime, while the plugin for your favorite build tool is under development, you still want coverage reports, so the solution is to use the command-line tool.
Maven is a tool that is widely used for building projects, with a lot of raw power under the hood. One of the most beneficial features of Maven are the plugins, which allow people to customize their builds however they see fit.
Saga has a Maven plugin that is available from Maven Central. This means that a default installation of Maven would immediately have access to the plugin, allowing for easy integration with any build. To configure the Saga Maven plugin, just add the following basic lines to their "pom.xml" file:
Gradle has been gaining popularity as a build tool, which is not surprising given that it allows for expressive configuration using a Groovy DSL, and because it enables using Groovy code in the build configuration descriptor. Thus, after a submitted user request, a Gradle plugin was born.
Gradle configuration is even simpler. This is what a typical "build.gradle" file might look like:
The command-line tool is basically a JAR with all the dependencies included. It works just as well as any other implementation, and using it is very simple:
Future of Saga
As of the time of writing this, the GitHub issue tracker for Saga counts 9 open and 68 closed issues, and 23 of the closed ones were requests for enhancements, which I always happily welcome. The currently open requests, among others, include:
- Branch coverage
- Running tests off a web server instead of the filesystem
- Report aggregation
There are many more improvements yet to come, and the only problem I am having is the lack of time to develop new features, so sometimes I have to concentrate on bugs and allow the features to brew for some time before implementing them.
So, I have to throw in a shameless plug and urge you to contribute to the project. There are a lot of things that could be added and improved, and the core instrumentation code is pretty stable, so you most likely would not have to touch it. Some people have already solved their and someone else's tickets, which is admirable and I am truly grateful for that.
However, the project needs more ideas and contributions from the community, so if you speak Java, please, do not hesitate to check out the code (there is not a lot of it anyway, nothing to be scared of), play with it and pick up a ticket to solve – or come up with improvements that you would like to see.
Apache Maven: http://maven.apache.org/
Maven Central Search Engine: http://search.maven.org/
Marat Dyatko's GitHub page: https://github.com/vectart
More Software Testing Resources
- Source Code Analysis Tools
- Software Testing Magazine
- Software Testing and Quality Portal
- Software Testing Tutorials and Videos