Bromine - Open Source Test Management Tool for Selenium

Rasmus Berg Palm, Jeppe Poss Pedersen, Visti Kloft, Bromine Foundation

This article is about Bromine, an open source functional web test automation tool for Selenium. We'll go through the background for Bromine and explain what Bromine is and why you should (or shouldn't) use it. Finally we'll introduce two real world scenarios in which Bromine is used.

If you have worked with test automation of web sites you have most likely heard of Selenium. For those of you who have not, Selenium is an open source test automation tool that has a lot to offer, in particular capture / replay functionality and means to run your test in a distributed environment. This is the foundation that Bromine builds on. In short Bromine can be described as a management tool for Selenium. It offers is a way to organize your test scripts and easily run them in different browsers and operating systems.

Web Site: https://sourceforge.net/projects/bromine/
Version Tested: Bromine 3 RC2
License & Pricing: GPL version 3 & Free

Once installed you can access Bromine through your browser. From here you can create new projects that can contain an array of requirements and test cases. For each test case you can upload a Selenium test written either in Java or PHP. Once you have uploaded the script head over to the Test Lab where you will be able to run either a single test case or a complete suite of tests which you have organized under a requirement. If you have the necessary capacity, the tests will be run in parallel saving you time compared to running them sequentially via the IDE or on a single machine through Selenium RC.

Figure 1 - Bromine showing the results of two test cases in seven OS/browser combinations

Background

So, how did Bromine come to be? If you have ever worked as a software tester you know how daunting it can be testing that the same menu link still works, build after build. This is where Selenium came into the picture. It is open source, so there was no need to invest in a proprietary automation tool which could easily have cost several thousand dollars per user in acquisition and then come the service agreements on top of that. However, we soon realized it was complicated to organize, run, and share tests between testers as Selenium did not have anything to offer on that part. We then began the development of what was then called TRM (Test Result Manager). This was back in 2007. From here on we kept adding features to TRM and then in early 2008 we decided to rework the whole thing and renaming the project Bromine. We decided on Bromine because it is the 35th atomic number in the periodical table next to Selenium. This phase of the development continued to about mid 2009 where we decided to completely rework Bromine again, this time around using a PHP framework (CakePHP) hoping that it would attract more people to contribute as it would be easier to adopt the programming style. This led to Bromine 3 Beta being released autumn 2009, which entailed a signification amount of improvements over the last installment of Bromine.

What is Bromine and why should I use it?

Bromine is not perfect. Actually, it's quite far from what we we would consider perfect. It will fit some organizations just great and others less so. We'll try to explain what the pros and cons of Bromine are in this section. But don't take our word for it; give it a try. It's free after all.

First off Selenium does functional web testing and Bromine makes using Selenium easier. Selenium is probably the most used functional web testing tool out there. It works by starting a browser, injecting it with a lot of JavaScript functionality and then executing JavaScript through the browser which simulates user actions. A command could be 'click("submit button")' or 'verifyElementPresent("some div")'. The words in quotation marks are what are called locators, i.e. some string that identifies a part of a web site. This is mostly element ids or xpaths. Selenium comes in a couple of shapes. The most used are Selenium IDE, a Firefox plugin that does record and replay, and Selenium Remote Control (RC), which is a Java server that listens for commands and upon receiving them opens and controls a browser and sends the results back for each command. Lots of more info on Selenium can be found at https://www.selenium.dev/

What Bromine does, and does well, is executing selenium RC tests, storing the results and presenting them in an easily understandable way, all as an easily accessible PHP/MySQL web application. This is the core functionality and this is what we've spent most of our time coding. The rationale behind this is that when we first started using Selenium RC, these were the most frustrating issues we faced. What good is a test if it's hard to run and the results are difficult to save and interpret? From this core functionality we have begun to develop the business side of Bromine; requirement and test case handling. This side is fairly bare bones at the moment, but we're planning on to focus our efforts in this direction over the next couple of releases.

A standard workflow in Bromine goes something along the lines of this: Management creates requirements and details which operating systems and browser combinations these should be tested in. Test manager creates test cases to cover these requirements. The tester records a script in selenium IDE doing what the test case specified. Tester exports the test script into Java or PHP and uploads it to Bromine. The tester selects the test case (or requirement, or even an entire project) and presses the run icon, which will launch the tests in the required OS/browser combinations (given that you have test machines setup having the required OS/browser combinations of course).

The tests are scheduled, much like a printer job queue and executes when the test machines are available, i.e. not running any other tests. The test script drives the browser and Bromine captures every command sent between the test script and the RC server as well as the status of the commands. This is then saved as a test run on the given test case and OS/browser combination. The test case will be flagged as passed or failed based on the results of the latest test run, and in turn any requirements the test case belong to will be flagged as passed or failed as well. The poorest result takes priority so that a single failed test case in a requirement with lots of test cases will mark the requirement as failed.

Bromine is unique in that it acts as a GUI for Selenium, closing the gap between technically minded tester and less technically minded management. In addition, it is free and open source. We don't feel that there are any other tools out there that fit this description.

Included in the Bromine RC2 release was Hudson and Sauce Labs integration, which are some features we're pretty excited about.

Figure 2 - Bromine with the scheduling pane open for a single test case

A real life scenario at Test Lab I/S

Test Lab I/S is owned by Visti Kloft and Jeppe Poss, two of the Bromine founders, and specialize in automated testing. The following is an example of why and how Test Lab uses Bromine, based on a couple of large projects. Unfortunately it's impossible to be more detailed due to NDA's with the customers.

How do Test Lab I/S use Bromine in the real world? Bromine helps us solve issues in big test setups. We have tested large web applications which included large number of sites, many localized versions, browsers, and large number of test cases.

In Bromine you organize your test cases and with one click you can start hundreds of test runs in different browsers. Imagine a project with a hundred test cases that needs to be tested in four browsers (FF3, IE7, IE8 and Safari) on three environments (test, QA and production). This gives 400 test runs each time we need to run our regression tests and that creates a few issues:

• Capacity issue (load balancing)
• Maintenance of the test scripts
• Overview of the current state of the application (does all the features work?)

Capacity issue

Running 400 test cases takes time and to solve this issue load balancing is needed. Bromine does just that. Bromine looks at the available Selenium RC (called nodes in Bromine) and will try to run as many tests as possible at the same time.

Maintenance of the test scripts

Having a hundred test scripts requires a lot of maintenance and instead of doing our own revision control you can just use SVN (or similar tools). All the test scripts in Bromine are in a certain folder and if you put that folder under revision control you can easily maintain your script without having to learn another new tool.

Overview of the current state of the application

Bromine gives us an excellent overview of the current state of the application. While it's not out there doing exploratory testing for us, we have it setup to test for known bugs as well as the core functionality of the application under test. Are all the pages online? Does each page links to the right pages? Does the rate, buy and signup functionality work? Having certainty on these issues frees up time for tasks humans are good at; finding tricky bugs and making developers miserable.

A real life scenario at Roskilde Festival

The following case study is a blog post by Anders Nickelsen, the leader of the QA-team in the IT-development group at Roskilde Festival, reproduced here with permission.

The organization

The Roskilde Festival located in Roskilde, Denmark, is the second largest music event in Europe and entertains more than 100.000 guests over 4 days (+4 day warm-up) annually. The festival is made possible by approximately 25.000 volunteers that work either before, during or after the festival. Amongst these are 40 volunteers working throughout the year to develop and operate 10+ specifically designed web-based information management systems used by the festival.

The challenges of volunteer developers

As with many open-source projects, the group of volunteer developers is very dynamic, since people can leave or join rather spontaneously. Therefore, a key element is to provide new developers a simple entry into the development process, and ensure that systems stay stable even if developers drop out.

Contrary to many open-source projects, developers are not end-users, so there is no obvious itch to scratch [1] until new developers have experimented with the systems and technologies. Also, once new developers have made contributions to the systems, they need to see it used by end-users in production soon after to avoid losing interest in the project.

Quality assurance by continuous integration

To provide sandboxes for experimenting and short 'time-to-production', we have built a versatile quality assurance framework to allow new developers to dive right into changing codes and try out new things, without the risk of breaking any of the running systems. Through a series of verification steps, both manual and automated, all system contributions are continuously integrated into the production mainline in a lean and reliable way.

The development process

Our setup builds around the philosophy that "developers need flexibility, operators need stability". To achieve this we use a chain of environments. As contributions flow through these environments - from the developers hand into production mainline - the developers flexibility becomes limited as focus is turned to stability of a release. The chain consists of the following 4 environments that are focused around per-system subversion repositories.

Development environments: VirtualBox images with replicates of the production environment that can be used locally by developers to try out new things. The developers commit contributions from the images into subversion projects.

Preview environment (alpha-test): Central servers with exports of the subversion projects that allow developers and change managers to check out subversion revisions to preview individual changes and run unit tests

QA environment (beta-test): Central servers with exports of the subversion projects that allow beta-testers to verify releases through integration tests and acceptance tests, and release managers to test release procedures. QA environments are separated from preview environments because they are also used for end-user education, which requires them to be more stable and less frequently updated.

Production environment: Central servers that host production versions of the systems, which are used by end-users.

The test framework

The test part of the continuous integration framework consists of the following tools:

• Hudson
• Selenium
• Bromine 3
• Sauce OnDemand by Sauce Labs

Hudson automatically triggers basic testing in the preview environment on every commit to Subversion. Commits are considered atomic and only include one change each, which is validated manually - also in the preview environment. Atomicity enables easy rollback of changes and preserves traceability.

Integration tests and end-user acceptance tests are recorded using Selenium IDE and collected in Bromine. Before the release of a new version, Hudson triggers Bromine to execute the entire test-suite of Selenium tests from the previous release as a regression test on the new system deployed in a QA environment. The tests are executed either on dedicated internal Selenium clients, or in the Sauce Labs cloud via Bromine's integration with Sauce OnDemand. Bromine produces result reports which provide an overview of the success rate of the tests. The result reports are automatically integrated into Hudson, which makes the test result overview fast and clean. Futhermore, end-user testers make exploratory testing in the QA environment to validate the released bug-fixes and features.

When the new release is approved, the Bromine test suite is updated to include new Selenium tests for the new bug fixes and features.

After release, the wheel turns once again with planning of a new release, where everyone - new as old - can participate, with small or big changes, thanks to the integrated quality assurance framework.

Conclusion

Bromine is an open source functional web testing tool. It is based on Selenium which is also open source and probably the most used web automation tool out there. Bromine's main strength is its ability to make the life of the Selenium powered tester easier. Bromine provides an easy way to organize and run your test scripts. It also gives you the opportunity to tie in the results of the test scripts with requirements and test cases. Bromine is a web application, which is easy to access and use for a large number of users. A headless Linux server with 256mb of RAM will probably host Bromine satisfactorily for an average small business (up to 5 users online simultaneous, more will require additional memory). Bromine integrates with some of the cool players in the industry like Hudson.

Bromine is not a magical tool which will make all your problems vanish in the blink of an eye. It's just a tool and tools are nothing without methods. You'll need to remember your ISTQB syllabus, and recall that automation is hard work and requires more abstraction than record and playback. It definitely pays off in the end though, as we hope our case studies have shown.

[1] The Cathedral and the Baazar, Eric S. Raymond, O'Reilly Media ISBN: 1-56592-724-

---

More Software Testing content

• Software Test Management Tools
• Functional Testing Tools
• Software Testing Magazine
• Software Testing Videos and Tutorials

---

Click here to view the complete list of tools reviews

This article was originally published in the Fall 2010 issue of Methods & Tools