Enterprise scale live device test automation for Android on Jenkins

"No darlin', don't make me explain it. I don't know why but I don't want to."

~ Ted Hawkins

Update: the included test system in Gradle supports parallelized test automation on all connected devices as of version 0.3. The current version is 0.7.3. See the following link. Obviously there is a lot of bluster about Google disrespecting scalable live device test automation which I could edit out at this point but A) Gradle Android support is not yet at v1.0 and B) that would be a wee dishonest. This is what I wrote. I was wrong. Check out Gradle. It is awesome-sauce! (edited: 2014-01-16)

Android's developer documentation is excellent and voluminous. There is an excellent mix of training, API guides, and JavaDoc references. So it piques my attention forcing me to ask "why" when a large topic I'm interested in is conspicuously absent. With Xcode 5 and Mavericks Server, Apple has made a serious play to support the enterprise continuous integration practices and live-device test automation for iOS. What system is provided by Google for their Android platform supporting scalable, live-device test automation? Clearly with its Java heritage with ages of continuous integration support, extensive command-line API in ADB, and massive JUnit library, all the pieces are there. But the documentation doesn't talk about scale, Ant and now Gradle can't handle multiple devices, the ADT plugin only automatically serializes test execution on multiple devices but that doesn't apply to the command-line tools. I could go on. So what gives? Why aren't the dots finally connected? 

I just assume that Google's massive, proprietary internal testing system is partly to blame. They can't package it with Android. But since it solves all the enterprise scale issues for them, the engineers don't prioritize solving that for others, or even just explaining it. Heck, despite all the reasons to loathe the emulator, Google not only leverages their massive server resources to reduce the pain of testing on emulators, they go so far as to decry live device testing for being expensive of all things. See their presentation and slides from GTAC 2013. To Google I say, "no, live device testing is irreplaceable." It is the foundation for one of my favorite robust and growing companies.

So now that we've covered the history and context, let's get down to brass tacks. My work over the last 3 years has included building a device lab, deploying a Jenkins CI system, and evangelizing its use among the various product teams here at the studio. I wrote an article on this blog touching lightly on the components of our system last year and the time has come to really lay out the details, particularly the crucial adaptation of Jenkins' master and slave node architecture which enables scalability through parallelization. I'm not going to write the book on how to set up Android test automation on Jenkins. That's already been done by Diego and it is a great resource. What I am going to do instead is the following:

1. Describe the node hierarchy and the method for configuring it in a device lab
2. Provide sample template jobs for building and testing an application which you can import DIRECTLY INTO YOUR JENKINS DEPLOYMENT (all caps!!!)
3. Provide an open-source release of the script we use to tie the whole system together

Jenkins-ci.org has good documentation on setting up distributed builds. They even go so far as to provide a step-by-step guide here. The key to understanding parallelization of Android test automation is to configure a slave node for each device you want to include in your test system. What makes this scalable is that you can host multiple slave nodes on the same computer. This means you can have a computer with an array of USB hubs and devices connected to it all talking to the same ADB service at the same time. There is an upper limit on the number of connected devices ADB can manage though I'm uncertain of what that number specifically is. I have heard it is on the order of 40 or 50. Let me illustrate what this can look like. Given 1 build server, 1 lab computer, and 3 devices connected to the lab computer via USB, we'll create 3 slave nodes.

• Build server
• hosts Jenkins process for master node
• Android SDK is installed for building projects via Ant
• Lab computer
• configured with DNS (or at least just static DNS) - e.g. 10.100.10.100
• supports SSH - e.g. devicetest@10.100.10.100
• Android SDK is installed for supporting device testing and app inspection
• Device A
• USB debugging enabled
• Device B
• USB debugging enabled
• Device C
• USB debugging disabled. Ha ha! Just kidding, it's enabled.

Basically once you get the pattern for the first slave node, you repeat it exactly for all devices on the lab computer only changing the Name and a device serial environment variable value. Here are the important config details to consider and sample settings:

• Name: "Device A"
• this will be used across Jenkins for linking jobs to the slave node under the "Restrict where this job can be run" setting on your test job
• # of executors: "1"
• this just ensures that only one test runs per device at a time, seems obvious but don't forget it.
• Launch method: "Launch slave agents on Unix machines via SSH"
• Host: "10.100.10.100"
• Credentials: "devicetest (10.100.10.100)"
• I'm using this as an example because of the config I found convenient and in this case "devicetest" is a user on the Lab computer with SSH access and credentials stored on the master node
• Environment variables
• name: "DEVICE_SERIAL"
• NOTE - in this case "DEVICE_SERIAL" is inherited by any job that is executed on that slave node's context (as opposed to the machine itself, hence running in parallel is threadsafe).
• value: "00000000000000A"

For each successive slave node config you can just copy the config for "Device A". Just make sure to update the name to reflect each new device and update the value for the DEVICE_SERIAL environment variable. This supplies the plumbing Jenkins needs to manage distributing your test automation in parallel across your slave nodes (even though technically it will be the same host computer for each one, trust me, it works in parallel).

To take advantage of this architecture, I recommend using individual test jobs for each node and separating the test job from the build job. If your build job triggers downstream test jobs, they can each run independently, fail in any unique way, and if you're really adventurous provide direct debugging per device within the Eclipse IDE using the Mylyn connector plugin. Follow me here, that means your developers can debug test results across hundreds of devices from their IDE as though the devices were connected to their development machine, all while offloading the processing and time cost of the test passes to the CI system.

Words, words words! Give me something I can use!

Because I'm a generous test monkey, I've gone ahead and created and tested two jobs in our Jenkins instance which perform what I've just described: an upstream build job and a downstream test job. To scale this, you simply duplicate the test job and point it at a second, third, forth, etc, slave node (making sure to modify the name appropriately because Jenkins needs unique job names). These jobs have been exported as XML via the Jenkins CLI tool. Using this same tool in your environment will allow you to reverse the process and import the XML as a new job using the following syntax:

java -jar jenkins-cli.jar -s http://server create-job newmyjob < myjob.xml

...where http://server is the web address of your Jenkins instance and myjob.xml is the name of the copy of my XML. It should even automatically link the two jobs which already identify their dependency in the XML. You can find the XML for each project in the GitHub repo at the end of this article. I'm running a Master node in Mac OSX Mountain Lion and a slave node running Ubuntu Linux 12.04 LTS. The configs include shell commands tailored for those environments. Please adjust as needed for your environment.

[edit] On the excellent suggestion from Christopher Orr, I've added a Jenkins Recipe to the repo which allows you to not only import the configs more quickly and easily than using the Jenkins-CLI (just make sure you install the plugin first), it will also import the plugin dependencies. That's some serious heavy metal stuff, Kohsuke. /horns [/edit]

Speaking of scripts, there's more...

Buried in the test job config is a reference to a script called "test_runner_script.sh". This script does a lot of really convenient things:

• Takes parameters from the test job identifying:
• Test ID
• Project APK
• Test APK
• Device ID
• Test Suite
• Test Runner
• Coverage Enabled
• It can handle screenshots from Robotium
• specifies directory to collect them in
• retrieves them for processing
• It can handle the XML output from the Polidea Instrumentation Test Runner (highly recommended for reporting)
• It can handle the Emma code coverage file output
• that's right, who needs Ant?
• note that the upstream build job is configured already to generate this for you
• It can uninstall any previously installed versions of the app based on package name
• It can accept all test suites supported by Android's InstrumentationTestRunner including annotated tests

That's a lot of extras to factor in so please take the time to CAREFULLY inspect what I'm doing in the build and test jobs I've exported. The best way to do so is after importing the XML, trust me. The XML is just fluff, the script is open source. Please feel free to adapt any of this to suit your needs and work best in your environment. 

Looking to the future, especially in the Android Test Kit, I'm hoping that an XML test runner version of the GoogleTestRunner can be developed (possibly by just importing the Polidea test runner's only class into the source for the GoogleTestRunner and extending it via the Test Runner Bridge; look at the source code for Espresso and Polidea's runners, you'll see what I mean).

Here's the repo:

https://github.com/RussellCollins/AndroidAutomationWithJenkinsParallelization

Have fun! Happy testing!