Everybody Tests

I first learned about OpenSTF from a tweet by Square's Jake Wharton . It's frickin' awesome. I believe I've posted about this before but seriously go check it out if you're an Android developer or tester who has a pool of shared test devices to work with. It's FOSS and boss. Go check it out. At the time of Jake's tweet, there was no integration between the awesome little project and my cluster of devices configured to host test automation in Jenkins. Because of the open source natures of both OpenSTF and Jenkins, and the robust plugin system on Jenkins, it seemed inevitable to me that a plugin would be written eventually to marry the two. I even speculated that I might write one if I got the chance. Luckily for everyone that wasn't going to be necessary . It's actually really straightforward to set up and easy to use but here are a couple of issues with work-arounds you might want to consider: Issue -The API key from OpenSTF for the plugin...

UI automation is like a dumb computer user; a really, really, profoundly dumb computer user. It will fail, it will be obvious that it is failing. But it won't always know immediately why. Sometimes just observing it's failures by means of a screenshot of the failure is all it takes. As you recall from my last post on the topic , screenshots on failure are a potent way to quickly reduce the time to answer the question of why a UI test case failed. Google's Android Test Tools team likes screenshots enough that they've included the capability inside the UiAutomator framework . Square likes it enough to have included it in their Spoon framework . And as we all remember, Robotium's been doing this for a long, long time . The main advantage to the approach I described previously is that screenshots would happen automatically based on errors of any kind in the test lifecycle. As of now, you're all probably migrating or have already migrated to the newer Android...

New Android build system and test runner, same goose chase using undocumented features and hacks Oh, you thought they were ready to support full enterprise scale CI just because they moved to Gradle? As I've posted before, I am a big fan of Jenkins. It is extremely flexible, open source, and supported by a staggering array of plugins actively developed by engineers running over 100,000 instances of the server worldwide. With it's distributed node model, you can even build your own device cloud for hosting enterprise-scale automation , economizing hardware investments by sharing resources across multiple projects as well as speeding up automation by parallelizing test runs. I had been using a Jenkins-based system in the past to support instrumentation automation with Robotium quite happily. For the last couple years however, my work hasn't required that as much and I've found myself doing a lot more manual testing and using UiAutomator which didn't require...

In 2013 at a Google-hosted event in New York called the " Google Test Automation Conference ", or GTAC for short, a team of Google engineers tried to sell the attendees the idea that a device lab consisting of real world devices is not as maintainable or optimal of a test bed as a vast server farm running emulators. That's right. Google said that in order to avoid the tyranny of dealing with the real world everyone should have a Google-scale server farm for testing their apps ( video , slides ). And then 2 months later they bought Appurify . Those are pretty epic mixed signals. Example of less paradoxical signaling. Now don't get me wrong, I would LOVE to have Google's server resources for my automated testing. But like many companies who do not make mobile apps or even software in general as their primary business, my current employer seems to prefer to invest more shall we say "modestly" in their app test automation environment resources. ...

Wow! It has been a long time since I've updated this blog; so long, in fact, that I a whole new version of Android has been announced (okay maybe 2 or 3). Like many of you, I was paying very close attention to the live streamed sessions at this year's Google I/O. At first I thought L contained nothing new for my QA interests but upon further digging I found a few tiny gems: "You can execute shell commands from your instrumentation test with the new android.app.UiAutomation.executeShellCommand(). The command execution is similar to running adb shell from a host connected to the device. This allows you to use shell based tools such as dumpsys, am, content, and pm." From the API Overview document for Android "L" Developer Preview Oh that's nice.  I guess that means I don't need these anymore: -     public static void runShellCommandStringArray(String[] command) throws IOException {        try {.. -   ...

"No darlin', don't make me explain it. I don't know why but I don't want to." ~ Ted Hawkins Update: the included test system in Gradle supports parallelized test automation on all connected devices as of version 0.3. The current version is 0.7.3. See the following link . Obviously there is a lot of bluster about Google disrespecting scalable live device test automation which I could edit out at this point but A) Gradle Android support is not yet at v1.0 and B) that would be a wee dishonest. This is what I wrote. I was wrong. Check out Gradle. It is awesome-sauce! (edited: 2014-01-16) Android's developer documentation is excellent and voluminous. There is an excellent mix of training, API guides, and JavaDoc references. So it piques my attention forcing me to ask "why" when a large topic I'm interested in is conspicuously absent. With Xcode 5 and Mavericks Server , Apple has made a serious play to support the enterprise continuous in...

"I'm looking over your shoulder... only because I've got your back." ~ Stephen Colbert After my recent UiAutomator review a user brought up an important question about the use of UiWatcher. The watchers serve as async guardians of the test flow, making sure the odd dialog window doesn't completely frustrate your tests. Having a tool that automatically watches your back when you're focused on the functional flow of your tests is awesome. 100% pure awesomesauce. Since the API documentation on watchers is scant and the UI Testing tutorial on the Android dev guide doesn't cover their use in depth, I figured I should add a post here that goes over a simple scenario demonstrating how to use this fundamentally important UI automation tool. In my example code below, I'm using uiautomator to launch the API Demo app (meaning run this against an Emulator built in API level 17 - I used the Galaxy Nexus image included in the latest ADT and platform tools). ...

"Drunkenness does not create vice; it merely brings it into view" ~Seneca So Jelly Bean 4.2 landed with much fanfare and tucked in amongst the neat new OS and SDK features (hello, multi-user tablets!) was this little gem for testers: UiAutomator.jar. I have it on good authority that it snuck in amongst the updates in the preview tools and OS updates sometime around 4.1 with r3 of the platform. As a code-monkey of a tester, I was intrigued. One of the best ways Google can support developers struggling with platform fragmentation is to make their OS more testable so I hold high hopes with every release to see effort spent in that area. I have spent a couple days testing out the new UiAutomator API  and the best way I can think of describing it is that Android's JUnit and MonkeyRunner got drunk and had a code baby. Let me explain what I mean before that phrase sinks down into "mental image" territory. JUnit, for all its power and access to every interface, e...

"A sub-feature of this is the screenshots feature. While most of the fare you will see will be OMG LOOKEY HEREs, with a little skill, you can take quite good images." ~PARDOX460, The Art of Taking Good Halo 3 Screenshots Remember back a few posts when I dropped a bit of info about maybe perhaps capturing screenshots on failed UI tests? Well it wasn't a lie and now I'm going to show you how. First, I'll go over the prerequisites. Second I'll present you with a code sample that demonstrates how to implement the screenshot tool in your tests. And finally I'll wrap up with some recommendations on further steps to embed this into your continuous integration environment and practice. Let's get started, shall we? Phase 1: Prerequisites The bell I'm forced to ring whenever it comes to Android test automation is that Google have deigned to build it around emulators. This is convenient for them because they're not beholden to manufacturers' en...