Zucchini - a Visual Testing Framework for iOS Applications

Vasily Mikhaylichenko

Zucchini is an open source visual functional testing framework for iOS applications based on Apple UIAutomation. It allows the usage of freeform text for interaction scenarios definition.

Web Site: https://github.com/playup/zucchini (not maintained anymore)
Version tested: 0.7.3
System requirements: Mac OS X 10.6, Xcode 4.2, Ruby 1.9.3, ImageMagick, CoffeeScript
License & Pricing: BSD, MIT licenses
Support: https://groups.google.com/d/forum/zucchini-discuss

Introduction

iOS mobile platform is uniquely consistent in terms of the operating system versions distribution [1] - as well as the devices range. With new iPhones being released once a year iOS developers get plenty of time to adapt to the new devices which also means more time to spend on polishing the actual applications. With iPhones only comprising 13% [2] of the overall smartphone sales Apple's AppStore still earns 2.6 times more revenue [3] than Google Play. Needless to say that iOS remains to be very appealing to developers.

With more than 900,000 applications [4] in the AppStore today developers try hard to make their applications stand out from the crowd. A well-crafted graphical user interface is an important aspect to this. Of course, not only the application needs to look and feel good, it also needs to be reliable and crash-free.

As Steve Jobs said,

"Design is not just what it looks like and feels like. Design is how it works."

Testing iOS applications

A great way of ensuring that your application is working properly as you continue the development is to maintain a comprehensive test suite. This normally involves at least two levels with Unit tests going hand to hand with the codebase and functional tests being at the higher (often business domain) level.

When it comes to iOS, the former category is represented with such tools as Apple's own XCTest [5], a popular Kiwi [6] framework, Specta [7] (all three based on the OCUnit framework), as well as Pivotal's Cedar [8] framework. As exciting as this category is, it is not going to be the focus of the current article.

Functional testing frameworks

Some of the popular functional testing frameworks for iOS applications include Apple's UIAutomation [9], KIF [10], MonkeyTalk [11], Calabash [12], Frank [13], Appium [14] and Zucchini, all quite different in terms of features and implementation. Below is a brief comparison between them.

Name	Scenario language	Step definition language	Assertion target	Communication protocol	Requires Xcode project modification
KIF	Objective-C	-	Accessibility labels	Same codebase as the application	Yes
Frank	Cucumber	Ruby	UISpec selectors	HTTP to a server embedded into the app	Yes
Calabash	Cucumber	Ruby	UISpec selectors	HTTP to a server embedded into the app	Yes
MonkeyTalk	Own DSL or Javascript	-	Own object model	UIAutomation	Yes
Appium	Any language with a Selenium WebDriver library		UIA elements	Selenium iOS driver	No
UIAutomation	Javascript	-	UIA elements	UIAutomation	No
Zucchini	Freeform	CoffeeScript	Screenshots and UIA elements	UIAutomation	No

Zucchini

Zucchini is a functional testing framework for iOS with a set of distinct features:

• Usage of freeform text for interaction scenarios definition
• Assertions based on screenshots
• Usage of CoffeeScript for screen contents description

Zucchini is based on Apple's own UIAutomation solution which means that:

• The underlying test runner is continuously improved and supported by the vendor
• It uses a very short communication path to a device or the iOS Simulator through Apple's proprietary Instruments [15] software
• It doesn't involve any modifications to the source code of the tested application

The framework is also compatible with continuous integration tools like Jenkins through exit codes and TAP-compatible [16] output.

Design

The mission of Zucchini is to enable efficient testing of iOS applications which is easy both to get started with and to support as the project grows.

A typical Zucchini test scenario looks like this:

Start on the "Books" screen:
 Take a screenshot
 Tap "Add"
 Then on the "New Book" screen:
 Clear the "Title" field
 Type "The Great Gatsby" in the "Title" field
 Type "F. Scott Fitzgerald" in the "Author" field
 Take a screenshot
 Confirm "Save"

To get it working, the application screens need to be backed up by CoffeeScript classes like this one:

class NewBookScreen extends Screen
  anchor: -> $("navigationBar[name=New Book]")
  constructor: ->
    super 'new-book'
    extend @elements,
    'Title' : -> view.tableViews()[0].cells()[0],
    'Author': -> view.tableViews()[0].cells()[1]

It's valid to say that the framework targeted at developers and test automation engineers familiar with the basics of JavaScript or CoffeeScript.

Zucchini gives its users the access to UIAutomation APIs which means that Apple's official documentation can be fully leveraged while working with the framework.

Workflow

Zucchini is different from the other iOS testing frameworks in terms of assertions it enables. Instead of forcing the user to drill through the UI elements hierarchy to retrieve a label or textual contents of a particular element, it performs visual comparison of screens against reference screenshots.

Why test against screenshots?

As I mentioned it at the start of the article, iOS as a platform is very consistent. Which means that there aren't that many devices to test on - and not so much difference in terms of visual representation of the same application on different devices.

Screenshots comparison enables testing of many things like:

• GUI elements alignment
• Visual content that can't be represented with text like graphics and photos
• Font faces

As you may think tests like these can be quite picky. And from our experience, this has been very useful.

That said, particular screen areas can still be masked away from the comparison.

Zucchini spots a difference between a bold and a normal font

A good Zucchini test suite is meant to give you a complete visual representation of your application.

Getting started

Zucchini is easy to get installed [17] provided the existing Xcode installation.

You can then start by generating a project scaffold:

zucchini generate --project /path/to/my_project

As well as a feature scaffold for your first feature:

zucchini generate --feature /path/to/my_project/features/my_feature

Another good starting point is the Zucchini demo [18] which includes an easy to follow setup.

Key concepts

Feature file

Zucchini feature files describe user interaction scenarios leading to a particular state in the application. Once the state is achieved it's a good idea to take a screenshot to verify it against the expectation.

Feature files have a *.zucchini extension and are written in a close-to-natural language. While conceptually similar to Cucumber's [19] Gherkin language, it doesn't involve starting each line with Given, When, Then or And.

Then on the "Welcome" screen:

  Take a screenshot
  Tap "Menu"

Continue on the "Menu" screen:

  Take a screenshot named 'menu-first-time'
  Swipe right
  Tap "About"

Screen class

The first line of each feature section corresponds to a screen. Each screen in the tested application should be backed up by a corresponding CoffeeScript class. These classes serve two purposes:

Providing an anchor element

This element is verified to make sure that the current screen is the expected one. It should be unique to the screen.

Definition of screen-specific elements and actions

Each line in a feature file corresponds to an action. Zucchini comes with a few standard actions but sometimes you might need to define a custom one to use the UIAutomation API directly:

class AuthorScreen extends Screen
 anchor: -> $("navigationBar[name=Author]")
 constructor: ->
  super 'author'
  extend @elements,
  'Author': -> $("textfield").first()
  extend @actions,
  'Scroll to "([^"]*)"$': (name) ->
     view.tableViews()[0].scrollToElementWithName(name)

By default, Zucchini tries to resolve the elements you specify in your steps (as in Tap "Menu") by their name. If this doesn't work, you'll need to provide an @elements entry with a finder function, like in the example above.

Fortunately, Zucchini integrates the Mechanic.js [20] selector engine as the $ variable which makes navigating the UIAutomation elements hierarchy easier.

Reference screenshot

Zucchini test report

Since Zucchini asserts the identity of screenshots, you'll need a set of reference screenshots to test against.

As you run Zucchini, the screenshots it takes are collected to the run_data directory inside the current feature directory.

You can then either copy them manually or use a subcommand to update all of your reference screenshots with the newly taken ones:

zucchini approve features/my_feature

Mask

Sometimes you'll need to exclude particular screen regions from being compared. These could be animated areas, keyboard layout switchers, etc. Zucchini lets you provide screen masks for these specific cases.

It applies the masks to both newly taken screenshots and the reference ones before the comparison.

Masks are solid-color transparent images matching the size of the screen

How it works

You can run an individual feature or the whole directory using the subcommand:

zucchini run my_project/features

Zucchini works by compiling all the CoffeeScript source files into a single UIAutomation-compatible JavaScript file which it executes on a device or on the iOS Simulator through the Instruments command line tool. As the script finishes, it uses ImageMagick to mask and to compare the screenshots.

If you choose not to take any screenshots in your feature, the test outcome will depend on whether the UIAutomation part finishes successfully. Thus, Zucchini can be used just as a convenience wrapper / test runner around UIAutomation and Instruments.

Summary

Zucchini is a powerful way of ensuring the visual consistency of an iOS application. It's a great choice if:

• You are using custom UI controls in the application, or
• Your application has got a significant amount of non-textual content, or
• You just want to give your UIAutomation tests more structure

The framework is a good starting point in functional testing of iOS applications. Simple to get started with, it should also appeal to developers who considered functional testing before but saw it as something too difficult or time-consuming.

References

1. Apple finally charts iOS fragmentation, and it puts Android to shame, http://venturebeat.com/2013/06/21/apple-fragmentation-what-fragmentation
2. Apple's iOS loses marketshare to Android, Windows Phone,http://articles.washingtonpost.com/2013-08-07/business/41155227_1_smartphone-market-ramon-llamas-new-iphone
3. Google Play's App Revenue Share Up 8% Since November, But Apple Still Leads,http://techcrunch.com/2013/05/29/google-plays-app-revenue-share-up-8-since-november-but-apple-still-leads/
4. Apple: 900,000 apps in the App Store,http://www.theverge.com/2013/6/10/4412918/apple-stats-update-wwdc-2013
5. XCTest, https://developer.apple.com/technologies/tools/
6. Kiwi, https://github.com/allending/Kiwi
7. Specta, https://github.com/specta/specta
8. Cedar, https://github.com/pivotal/cedar
9. UIAutomation, https://developer.apple.com/library/ios/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html
10. KIF, https://github.com/kif-framework/KIF
11. MonkeyTalk, http://www.gorillalogic.com/monkeytalk
12. Calabash, http://calaba.sh/
13. Frank, http://www.testingwithfrank.com/
14. Appium, http://appium.io/
15. Instruments, https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/Introduction/Introduction.html
16. Test Anything Protocol, http://en.wikipedia.org/wiki/Test_Anything_Protocol
17. Zucchini Installation Notes
18. Zucchini demo project
19. Cucumber, http://cukes.info/
20. Mechanic, a CSS-style selector engine for iOS UIAutomation, https://github.com/jaykz52/mechanic

---

More Software Testing Knowledge

Functional Testing Tools

Software Testing Magazine

---

Click here to view the complete list of tools reviews

This article was originally published in the Fall 2013 issue of Methods & Tools