Lies, Damn Lies, and Android Non-Rooted Device Code Coverage

"There are three kinds of lies: lies, damn lies, and statistics."
~Mark Twain, aka Samuel Clemens

In a previous post I mentioned rather blithely that we were collecting code coverage data from live devices using Emma. If you're unfamiliar with Android's use of Emma, which comes with the Android SDK tools when you install them, the documentation won't do you much good. It isn't to say that the Android developer portals' "Testing from Other IDEs" section isn't helpful or that the online documentation for the SDK's InstrumentationTestRunner isn't either. Far from it. But if that's all you read, you may find yourself getting confused shortly after getting started.

Let's say for example, that you begin your deployment of code coverage tracking via the robust set of Ant commands included in the default build.xml. You may notice from the Testing from Other IDEs section above that there is a simple command you can run that allows you to also get code coverage data from your build and subsequent test jobs. If you attempt to run this on a device, via the "ant emma debug installt test" command the process will run smoothly until you get a strangely unforeseen message aka "the big emma lie":

WARNING: Code Coverage is currently only supported on the emulator and rooted devices.

The folks on Stack Overflow have tackled this to some success and so have the folks in the Android Developers Google Group. They're smart people and very clever but all of them seem to get hung up on that big emma lie. You may be asking, "what's the lie? It works when I use it on an emulator and doesn't when I use it on a non-rooted device. This seems consistent." Well, I'll tell you but first a little story about JUnit XML output in Android.

In Android's security paradigm, the sandboxing of an application to writing to its own directory keeps non-rooted devices under control. Files installed by your application including databases, images, etc all wind up in '/data/data/com.your.application/'. That is true of code coverage output and even XML output if you're using Polidea's version of the InstrumentationTestRunner. I mentioned our use of this third party JAR in a previous post because it is great for getting standard XML output from Android's JUnit tests.

Running continuous integration on live devices would be kind of pointless if you couldn't pull all your test results back from the device. Hence why Polidea named their project the "missing xml junit test runner". It is something folks in the Java development community outside of Android have enjoyed for years so what's the issue with Android? I don't know. All I can assume is that the development model focuses on the use of emulators and Eclipse. This is why using the command line to execute test automation is included in the Android Developer Portal's section on "other IDEs". It is non-standard. So non-standard in fact that simply including the Polidea JAR does not grant you access to your XML on devices either. It works fine with emulators but like everything else in your project, it is sandboxed. The question is how can we get out of the sandbox without rooting the device or resorting to emulators?

At first we got clever. We flexed some coding skills and adapted the open source Polidea JAR to fork the test data output stream into LogCat. We added a script to parse LogCat for specific tags and collect those parses into XML on the USB host server in our lab. We thought we were quite clever with this because it got us our nice, portable JUnit output off live devices and that was our first goal. It worked very well, despite feeling a little TOO hack-ish, for about a year. But when I started to revisit the purpose of the lab and testing on devices, I couldn't shake the feeling that we missed something fundamental about the Android sandbox. Turns out my hunch was correct and I cracked the lie of XML and code coverage from live, non-rooted devices. It was right in front of me all along.

Elsewhere in our build process, we had previously included a step on the Jenkins job to update the AndroidManifest.xml with a build name that made our test builds more human friendly. This simple method of inserting code into the build at compile time became the model for how we subverted that lie about emulators and rooted devices. Because we were doing this code insertion at build-time, nothing changed in the source code for the project and outside of that specific job. That same manifest also controls the scope of the sandbox by the 'uses-permission' flag. That flag is what I had previously experimented with in order to get screenshots from devices using Robotium (expect a post on this topic soon). And because I am lazier than clever, I didn't immediately at that minute use the same flag to push XML files and code coverage output to a shared location on the device sdcard.

It took about 2 days instead.

So there you have it really. Without making this into a step-by-step description of the whole process of getting code coverage data off live devices (I can include that in a different post if desired), the secret was using a specific Jenkins build job step to insert the following line into the application's AndroidManifest.xml

"uses-permission:android.permission.WRITE_EXTERNAL_STORAGE"

Now suddenly I could get all the data that the emulator-driven paradigm gave but instead from every Android device in our lab, simultaneously even. All I did afterwards was update the test runner script on the USB host server in our lab to point the XML output, screenshot output, and Emma code coverage output to a single directory on the device sdcard that gets erased after every run. YAY! Ponies for everyone!

A word of caution though. While you may find this unlocks a whole new world of device automation coverage (both the code and the platform diversity), you should be aware that Code Coverage data is a form of statistical data. Code coverage, like any statistic can tell a useful story but be careful to understand the meaning of the data in context so that you aren't trapped in "give me 100% coverage on all automation" Hell. The proper use of code coverage statistical data is beyond the scope of this post but keep in mind the old saying, "there are lies, damn lies, and statistics."