Everybody Tests

The rent is too damn high. Okay, let's look at the topic of device clouds. Microsoft's recent purchase of Xamarin means that now Google, Amazon, and Microsoft (3 major cloud computing service providers) all have their own device clouds which they also lease time on to the public (well, Google's purchase of Appurify hasn't resulted in that yet but it's coming soon and is currently in Beta). That makes 3 major cloud players who also build mobile apps buying their own device clouds and renting their overhead to the public. To me that's purely confirmation that if you intend to use real devices and you're a developer or a company that does a lot of development and testing of mobile apps, particularly on Android, you had better have your own cloud. If it were cheaper to rent than to own, why would those three major cloud providers buy existing device cloud as a service platforms? In reality, Google Ventures was the primary early investor in the Y-combinat...

"Never without my permission" ~ Leeloo (translated) android.permission.WRITE_EXTERNAL_STORAGE As a tester, you probably have a favorite app, maybe a favorite feature, possibly even a favorite bug. But you probably don't have a favorite permission. I've been taking advantage of it for years to do things like gather code coverage on non-rooted devices, logcat output from an entire test run, and probably my favorite usage is screenshots triggered when tests fail. Yep, that little permission has been a big pal of mine for a while. So it should be no surprise that I felt a little betrayed when I recently discovered a problem with it on devices running Android 6.0 aka "Marshmallow". Among the many changes brought about by the 6.0 release was a long-overdue overhaul to Android's permission scheme, notably, from an all-or-nothing approval of permissions at install-time model from Android 1.0-5.1.1 to the more nuanced permissions requested when requ...

When you talk to the platform tech lead for your project about how to coordinate what gets automated with what is tested manually and you're pretty sure he's convinced the manual testers treat each new build all like... When you haven't participated in the sprint demo phase of your team's bi-monthly review ritual and they suddenly ask for a demo of the test automation project and you're all like... When the senior manager suggests contrary to your professional advice that the right approach is to just get something running on this new project, even if you know it is broken, because you'll magically have time to refactor it later and do it right, so that you can just show some pretty graphs and you're all like... When your new project is so stuffed full of libraries for the sake of libraries that your debug build is failing on the dex limit, forcing you to ask the developers to make adjustments and they come back with a solution that you just KNO...

Take a minute and read the following 2-pager entitled " Guest View: Why developers shouldn’t perform software testing ". Man, I tried to like this article because the author, Martin Mudge, clearly knows that businesses who undervalue quality by trying to eliminate testing through simply shuffling the traditional testing task to developers are making a huge mistake. Unfortunately he begins by making an assertion that testing overburdens a developer which at face value is complete nonsense. If you feel overburdened, it is a timeline issue and that is true no matter WHAT the nature of the tasking is.  So his assertion of “one the most important” contributing factors being overburdening the developers is massively flawed. It gets immediately worse from there because his second point is about time constraints.  Mr Mudge just gets so much wrong. What he really should be shooting down is the idea that testing, as a cost-center not as a task, can be eliminated by having your p...

UI automation is like a dumb computer user; a really, really, profoundly dumb computer user. It will fail, it will be obvious that it is failing. But it won't always know immediately why. Sometimes just observing it's failures by means of a screenshot of the failure is all it takes. As you recall from my last post on the topic , screenshots on failure are a potent way to quickly reduce the time to answer the question of why a UI test case failed. Google's Android Test Tools team likes screenshots enough that they've included the capability inside the UiAutomator framework . Square likes it enough to have included it in their Spoon framework . And as we all remember, Robotium's been doing this for a long, long time . The main advantage to the approach I described previously is that screenshots would happen automatically based on errors of any kind in the test lifecycle. As of now, you're all probably migrating or have already migrated to the newer Android...

When you go from building UiAutomator 1.0 tests in Eclipse as a stand-alone JAR over the last couple years to trying to use UiAutomator 2.0 within a product codebase with Gradle and nothing looks familiar... When you figure out why you were getting a null pointer exception after 2 days and it was all because of a copy-paste error... When you start trying to explain a tricky problem with a new test framework to the front-end developers on your project and it starts to feel like they're more spectators then collaborators... When you right-click on a test method and choose "Debug as..." and start debugging your UiAutomator test after all those months of needing to compile a JAR and edit a shell script prescribing which test case to run instead... When you can no longer use all those fantastically useful shell commands you were able to run in your UiAutomator 1.0 tests because 'adb shell uiautomator...' imparted sufficient permissions but 'adb...

When you keep ramping up the event total count on your stress test because it mocks you by not resulting in crashes... When you join a legacy project and upon looking through their Gordian-knot of build dependencies realize that 90% of it is obsolete, yet subtly interdependent experimental code no longer included in the app (EWWWW)... When the project planners lowball the QA estimates because all they're doing is "throwing on a fresh coat of paint"... When that other team throws yet another party celebrating a routine bug-fix release...