JavaScript Code Coverage Analysis with Saga

Timur Strekalov, Zimory, http://zimory.com/

Saga is an open-source tool designed for JavaScript code coverage analysis. Written in Java, it aims at enabling easy and straightforward integration with various build cycles, providing support for different build tools, such as Maven and Gradle, and a command-line implementation mostly as a fallback solution when neither Maven nor Gradle are a good fit for your Continuous Integration environment.

Web Site: http://timurstrekalov.github.com/saga

Version tested: 1.3.3
System requirements:
Java 6+ for the core and the CLI tool
Maven 2.2.1 or 3.0.3 for the Maven plugin
Gradle 1.2 for the Gradle plugin
Any modern browser (Google Chrome, Mozilla Firefox, Safari) for viewing the HTML reports
License & Pricing: Open Source, Apache License, Version 2.0, Free
Support: Issue tracker at https://github.com/timurstrekalov/saga

Introduction

Writing automated tests is an essential part of software development, and unit testing the simplest and fastest way to prove the validity of your code. Since unit tests are designed to verify the correctness of specific units of your application in separation (such as classes in object-oriented programming), it is rather beneficial to know how much of the given unit is actually covered by the test.

For example, if your JavaScript function has 30 statements in it and you are writing a unit test for this function, it is very useful to get a visual representation of the statements currently covered as well as missed ones. Not only you would see that the current test covered only 24 of the statements, giving you 80% code coverage, but you could actually see the statements that were missed because of the code paths that the test has chosen.

Some teams establish certain standards for their code, setting a percentage of the code that must be covered by unit tests (e.g. 90%), and code coverage tools such as Saga allow them to track the quality of their code. Moreover, some teams configure their Continuous Integration servers, such as Jenkins, to fail the build if it does not meet the acceptable code coverage threshold.

Design principles

Saga was envisioned and designed from ground up as a tool that would allow people of very different skill sets, not only software developers, to customize their build cycle and use the appropriate implementation to suit their needs, without too much effort. It is highly configurable, but strives to provide sensible defaults for most of the parameters, requiring only three configuration items to be specified in order to produce a coverage report.

Key features

Dynamic instrumentation

One of the biggest advantages of Saga is that it instruments your JavaScript code on the fly, as opposed to doing static instrumentation first and then running the tests against the instrumented source files. This saves the user a lot of trouble trying to figure out what they have to configure and run, and alleviates the pain of understanding what instrumentation is all about. All the user has to do to generate a code coverage report is:

• point Saga to the base directory of their source codebase, such as "src";
• provide at least one name pattern of the tests to be run, such as "**/*Test.html;
• specify the output directory for the coverage report, such as "target/coverage"

That's it. Three very basic things one would want to configure anyway, and it is enough to free the user of the worries of the details such as instrumentation and let them have what they want.

Two types of coverage reports

Saga can generate two different types of reports:

• Total coverage report – a coverage report indicating all statements that were covered by all the tests that had been run.
• Per-test report. Unit tests are designed to cover a specific unit of your code, and it is very desirable that the coverage of one unit would not be affected by another unit's tests. While a total coverage report can show that a unit has 100% coverage, it might mean that it was referenced by tests that are not really testing this particular unit – it might even be possible when there are no tests for that unit.

A per-test report shows only the coverage of the code by each unit test. This is why it might be useful for determining whether the test that was designed to validate a unit is actually properly covering it.

Accurate results

One of the problems with generating a JavaScript code coverage report is that there is a big difference between what code is being loaded during a test run and what code needs to be tested, ergo, what code needs to be included in the report. It might be absolutely possible for a codebase of 100,000 lines of code that has one unit test to get a result of 100% code coverage from the single test if we only consider the code that was in any way referenced during the test.

This is exactly why Saga allows you to configure the code that you would like to have covered in order to discover the units that remain hidden from all your unit tests, allowing for the most accurate code coverage results.

Testing framework-agnostic

Even though Saga provides an example on how to integrate it with a testing framework for just Jasmine (in fact, using the jasmine-maven-plugin, to be precise), it can work with pretty much any testing framework that runs the JavaScript tests using HTML "runners" – files that wire up the code of the framework itself together with your tests in order for the latter to be executed from within the browser.

There is an idea to allow running testing scripts directly, but this feature is currently not supported in version 1.3.3. Saga can only execute tests that you would be able to execute using a normal web browser.

High Performance

Saga is by default configured to execute tests concurrently, making use of all of the computing power available in the machine. Instrumentation itself takes a fraction of the time, as does the fact that instrumented code gets somewhat slower due to the extra statements that are the core of code coverage generation being introduced into the code; all of this adds up, so making use of all of the cores of the CPU makes the whole process much faster.

Neat coverage report

By default, Saga produces two different types of coverage reports: one in HTML and one in a raw LCOV format, which can be read by Sonar, for example. The former, however, is a very slick interactive report that shows various code coverage statistics and allows you to see the actual statements that were covered and missed. It was designed by Marat Dyatko, an engineer with a great taste (when it comes to design), to whom I owe a debt of gratitude for taking the time to make the HTML report look as great as this:

Starting with version 1.3.3, Saga also provides support for generating PDF and CSV reports in addition to the HTML and LCOV formats.

Implementations

Core

Saga integrates with build tools through a plugin system, so you should be able to get all the features with any implementation that fits your needs. Saga currently has implementations for two major build tools, Maven and Gradle.

If you are using any build tool other than Maven or Gradle it might seem like you cannot use Saga, which, luckily, is wrong. First of all, you should consider submitting a feature request by going to the GitHub page and creating a new issue, or better yet, if you have experience with the concerned build tool and have the time to contribute an implementation, I will be happy to help you with it. In the meantime, while the plugin for your favorite build tool is under development, you still want coverage reports, so the solution is to use the command-line tool.

Maven

Maven is a tool that is widely used for building projects, with a lot of raw power under the hood. One of the most beneficial features of Maven are the plugins, which allow people to customize their builds however they see fit.

Saga has a Maven plugin that is available from Maven Central. This means that a default installation of Maven would immediately have access to the plugin, allowing for easy integration with any build. To configure the Saga Maven plugin, just add the following basic lines to their "pom.xml" file:

<plugin>
<groupId>com.github.timurstrekalov</groupId>
<artifactId>saga-maven-plugin</artifactId>
<version>1.3.3</version>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>coverage</goal>
</goals>
</execution>
</executions>
<configuration>
<baseDir>src</baseDir>
<includes>**/test/javascript/**/*Test.html</includes>
<outputDir>${project.build.directory}/coverage</outputDir>
</configuration>
</plugin>

Gradle

Gradle has been gaining popularity as a build tool, which is not surprising given that it allows for expressive configuration using a Groovy DSL, and because it enables using Groovy code in the build configuration descriptor. Thus, after a submitted user request, a Gradle plugin was born.

Gradle configuration is even simpler. This is what a typical "build.gradle" file might look like:

Buildscript { 
    repositories { 
        mavenCentral() 
    } 
 
    dependencies { 
        classpath group: 'com.github.timurstrekalov', 
                name: 'gradle-saga-plugin', version: '1.3.3' 
    } 
} 
 
apply plugin: 'saga' 
 
saga { 
    baseDir = new File('src')
    includes = '**/test/javascript/**/*Test.html'
    outputDir = new File('build/coverage')
}

Command-line

The command-line tool is basically a JAR with all the dependencies included. It works just as well as any other implementation, and using it is very simple:

java -jar saga-cli-1.3.3-jar-with-dependencies.jar \
	--base-dir src \
	--include **/test/javascript/**/*Test.html \
	--output-dir target/coverage

Future of Saga

As of the time of writing this, the GitHub issue tracker for Saga counts 9 open and 68 closed issues, and 23 of the closed ones were requests for enhancements, which I always happily welcome. The currently open requests, among others, include:

• Branch coverage
• Running tests off a web server instead of the filesystem
• Report aggregation

There are many more improvements yet to come, and the only problem I am having is the lack of time to develop new features, so sometimes I have to concentrate on bugs and allow the features to brew for some time before implementing them.

So, I have to throw in a shameless plug and urge you to contribute to the project. There are a lot of things that could be added and improved, and the core instrumentation code is pretty stable, so you most likely would not have to touch it. Some people have already solved their and someone else's tickets, which is admirable and I am truly grateful for that.

However, the project needs more ideas and contributions from the community, so if you speak Java, please, do not hesitate to check out the code (there is not a lot of it anyway, nothing to be scared of), play with it and pick up a ticket to solve – or come up with improvements that you would like to see.

References

Apache Maven: http://maven.apache.org/

jasmine-maven-plugin: http://searls.github.com/jasmine-maven-plugin/

Maven Central Search Engine: http://search.maven.org/

Gradle: http://www.gradle.org/

Groovy: http://groovy.codehaus.org/

Jenkins: http://jenkins-ci.org/

LCOV: http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php

Sonar: http://www.sonarsource.org/

Marat Dyatko's GitHub page: https://github.com/vectart

---

More Software Testing Resources

• Source Code Analysis Tools
• Software Testing Magazine
• Software Testing and Quality Portal
• Software Testing Tutorials and Videos

---

Click here to view the complete list of tools reviews

This article was originally published in the Spring 2013 issue of Methods & Tools