UiAutomator.jar: What happened when Android's JUnit and MonkeyRunner got drunk and hooked up

"Drunkenness does not create vice; it merely brings it into view" ~Seneca

So Jelly Bean 4.2 landed with much fanfare and tucked in amongst the neat new OS and SDK features (hello, multi-user tablets!) was this little gem for testers: UiAutomator.jar. I have it on good authority that it snuck in amongst the updates in the preview tools and OS updates sometime around 4.1 with r3 of the platform. As a code-monkey of a tester, I was intrigued. One of the best ways Google can support developers struggling with platform fragmentation is to make their OS more testable so I hold high hopes with every release to see effort spent in that area. I have spent a couple days testing out the new UiAutomator API and the best way I can think of describing it is that Android's JUnit and MonkeyRunner got drunk and had a code baby. Let me explain what I mean before that phrase sinks down into "mental image" territory.

JUnit, for all its power and access to every interface, every service, every nook and cranny in your app, is limited to only running within your app's context. It carries the full heft of all the years of *Unit development elsewhere in test automation but it remains too close to its unit test roots to be serviceably fast and agile to develop in for UI automation. Moreover the limitation of only being able to test the associated product app prevents testers from fully exploring the interactions between applications and systems elsewhere on the device.

Conversely Android's MonkeyRunner is based in a Java adaptation of Python (Jython, no kidding) and it has access to a whole host of interfaces outside of the particular application you're trying to test. It can be used for mobile web testing, capturing screenshots of any app interaction, and since it is based in Python, is fairly quick to script. Its shortcomings are pretty severe when it comes to deep integration with application architecture so you are left clicking sometimes blindly in order to generate events (sendkeys, x/y coordinates, d-pad actions, oh my!).

It is easy to see where these two tools could complement each other well if combined. Deep application integration with simpler scripting and full-device access, not just sandboxed inside a single app's context could be really powerful. I say "could be" for a reason though and that's after a few days' mucking around with the tool. Let's go over the good aspects first.

The Good:

1. Since UIAutomator still uses JUnit as the basis for its test runner, most of the familiar test case structures are available in all the best ways. 
2. Of special importance are UiWatchers which are like async police whom you can configure to lurk outside of test cases to catch common difficulties affecting tests (such as dialogs and alerts) or embedded within tests themselves for more specific triggers.
3. The XML hierarchy dump tool in Monitor/DDMS is amazingly fast by comparison to the old Hierarchy Viewer, and gets you everything you need as a tester to identify the specific UI elements your test will need without the distractions. Brilliant.
4. The tests compile as a separate JAR which you push to the device/emulator in shared filespace so that the application-under-test-sandboxing of former JUnit test projects will be a thing of the past. Even better, the JAR still executes on the device allowing for massive parallelization just like before (sure, I am tempted to brag about having a parallelization problem but I'll take the high road this time)
5. Simple, repeatable syntax for getting objects from the UI to interact with means you spend less time coding and more time constructing useful tests (at least, that's the hope, when it works)
6. Use of Accessibility labels enforces good coding practices. Just like iOS's UI Automation, this tool takes advantage of some oft-overlooked aspects of complete code and so testers get convenient UI hooks, and the sight-impaired get better apps. Win, meet my friend Win. You two have lots to talk about.

The Bad:

1. Just like the former JUnit, this one lacks portable XML test results output which means it is just feels like Google don't like good, thorough reporting. 
2. Furthermore, on the reporting side of life, the lack of XML output is compounded by the lack of Eclipse integration for running the tests. You will spend a lot of time with the command line. As you're aware from my previous posts, we've built an extensive CI system and hooked it up to a device lab which is accessible even from within Eclipse. This tool is not there yet.
3. JUnit's overly verbose coding style is still present meaning writing tests is still complex and you need to know a lot about device limitations, timing of events in the UI, and other kinds of non-trivial, deeper than scripting, test automation heavy lifting. I would say this is still a 4 out of 5 in terms of code complexity. Maybe a 3 if your app plays nicely. I would say my UIAutomator test cases are likely to contain 80% to 120% as many lines of code as my straight up JUnit-based UI test cases. No real gains here (yet).
4. This tool only works on devices and emulators graced with Android 4.1 (or higher). This is because the tests are run via an app included with the OS on the device. Over the next few years this will allow you to begin getting wider device coverage with tests written using this tool but until then you can't use the UIAutomator for compatibility tests. Fragmentation's a mean ol' dog who won't be put down easily.

So what are we left with when we add this all up? I'd say while UIAutomator won't revolutionize the way I write UI test automation overnight, it shows tremendous promise. When this little code baby matures over the next year or two, with some more support from the Open Source community, I can see this going way beyond tools like Robotium. For those of you who are very comfortable in Java and JUnit, this will get you writing UI Automation faster than Robotium did, and with its ability to break out of the sandbox, you'll be getting more creative with what you're willing to test. 

What's the TL:DR?

Why you want to use this: no more app sandbox for JUnit-based tests, better coding practices, more automatable scenarios with deeper device integration.

Why you'll want to pull your hair out: still no native XML output, no Eclipse integration, still too verbose to be clean and fast, and it only works on devices/emulators running Jelly Bean 4.1 and above.

Final Words

Writing comprehensive automation for UI tests on devices is still really hard. There is huge support in the Android community for brave testers who like a challenge and Google have provided us another option. If Android's JUnit and MonkeyRunner's drunken tryst resulted in all this already, I say we all buy Google's Android test developers another round.