Automatic UI testing on real devices - Android, Espresso 2.0 and AppThwack

What if you could develop a set of UI tests locally, and once happy, run them in parallel on a multiple Android devices somewhere in the cloud. Sounds nice. Let me explain how you can achieve this by combining Espresso 2.0 and the cloud service AppThwack.

To help you get going, I've created this gihub repo that shows you how.

And here are the test results you get from a successful run on 10 devices:

 

Now let's go through how you get this working.

Step 1: Create an app

I'll assume you already have this. If not just fire up Android Studio and go through the steps to create one and look at my initial commit.

Step 2: Add Espresso 2.0 tests

I followed the steps here, but I'll go a bit in detail.

Basically this all resulted into the second commit but here goes:

  1. add testInstrumentationRunner 'android.support.test.runner.AndroidJUnitRunner' to your defaultConfig
  2. add exclude 'LICENSE.txt' to packagingOptions (otherwise you get some conflicts because of duplicates)
  3. add androidTestCompile 'com.android.support.test.espresso:espresso-core:2.0' and androidTestCompile 'com.android.support.test:testing-support-lib:0.1' to your dependencies
  4. add a test to your project (Ctrl-Shift-T in Android Studio / Windows)
    1. I selected JUnit4
    2. Name it something
    3. Skip superclass and hit OK
    4. you now have a test src/androidTest/YourTestClassName.java
    5. Make it extend ActivityInstrumentationTestCase2<MainActivity>
    6. Implement the default constructor and a simple test case
  5. Run the test by right-clicking on androidTest/java in Android Studio and selecting "Run All Tests"

You are now ready to run this test on a remote device.

Step 3: Add Appthwack

You need an account on AppThwack. They have a free one that gives you 100 device minutes, after that you've got to pay.

Once your account is setup create a new Android app project called "AutomaticTesting", and go to your profile page and create and copy your API key.

 Now basically do the changes I did in my third commit.

  1. downgrade to Gradle 1.0 as the AppThwack plugin doesn't support 1.1
  2. add apply plugin: 'appthwack' to your root build.gradle file
  3. add an appthwack configuration to your app build.gradle and:
    1. add your apiKey = 'your_key'
    2. set type = 'junit'
    3. set project = 'AutomaticTesting'
    4. set devicePool = 'Top 10 devices (10)'

You are now ready to run the test remotely. On the command line write:

gradlew appthwackUpload

On a successful run it will say something like:

:app:assembleDebugTest UP-TO-DATE
:app:appthwackUpload
[AppThwack] Using Project 'AutomaticTesting'.
[AppThwack] Using DevicePool 'Top 10 devices'.
[AppThwack] Uploading apk 'app-debug.apk'.
[AppThwack] Uploading test apk 'app-debug-test-unaligned.apk'.
[AppThwack] Scheduling 'junit' run 'app-debug.apk (Gradle)'.
[AppThwack] Congrats! See your test results at https://appthwack.com/project/automat.....

You can now wait for either an email of the results, or go to the url above if you want to follow the test live.

Conclusion

It's actually quite simple to get this stuff up and running, at least now that I've found out how. But it's not all happy days from here on. Here are some problems I've run into:

  1. The docs for AppThwack and Espresso are out of date, hidden or non-existing. The best sources are here and here. But basically there are no proper in depth documents with best practices or guides on writing proper tests neither with Espresso or on / for AppThwack. Updated 13.03.2015: A newer version of the Testing Support Library was just released with updated docs - worth looking at.
  2. AppTwack doesn't allow you to require Google Play Services, so many tests will fail because the device doesn't have it or it's out of date. Updated 13.03.2015AppThwack is considering adding this filter.
  3. AppThwack doesn't let you specify location in the configuration (you can specify it in the manual runs). Neither can you specify locale. Updated 13.03.2015You can actually set this programmatically via their REST API but it's not as easy as I'd like.
  4. AppThwack doesn't take a screenshots when a test fails. This means you only have the logs to debug from. This makes it really hard  to find out whats wrong. Updated 13.03.2015: I found out that Robotium can take screenshots for you, and that AppThwack will fish these out and display them for you. See this commit for the details.
  5. AppThwack doesn't support turning off system animations. At least it doesn't document how to. Espresso comes with hints on how this can be done but neither of them document how to set this up in your own testing system.

At the moment I'm still on the fence if this can be a proper test setup that delivers reliable results. There is too much magic going on that I don't feel in control of. At times it feels like you're debugging the Mars Rover.

Update 13.03.2015: With my recent discoveries, I'm more much positive and believe this can result in stable auto-testing on real devices.

Btw. I've also discovered the Spoon library from Square (I'm starting to wonder if Jake is a cyborg sent back in time to help humanity with Android libraries) that seems to do what AppThwack does, but on all your local devices. In short: If you hook up all your real Android devices and start up all the emulators your machine can handle, you can run tests on all of them in parallel using Spoon. Sounds pretty neat too.