Is it the app or the server?

When an app stops working, or responds slowly, the app developer is always the first to get blamed. This is very frustrating when it turns out to be caused by a dead or unresponsive server and not the app itself.

The best way to determine blame is to set up some automated testing solution that simulates network calls from the app to the server and measures up-time and response time. Then, next time someone comes blaming you for making a crappy app you can refer them to the last days of test data and hopefully figure out where the real problem lies.

Today it's fairly easy to setup such an automated testing solution. Two alternatives we've tried are Pingdom and Runscope.

Pingdom

Pingdom is really a website monitoring tool but can be bent into shape for our purposes. To use it you add a URL as a Check and tells it how often it should ping this URL. It measures uptime and response time and does this from servers in US and Europe. It does HTTP, HTTPS, GET and POST and you can check for certain strings in the reply as part of the test. It can also send out alerts if the server is down for a certain period of time.

Shows up-time and response time for one of the Checks for one of the apps we develop

Shows up-time and response time for one of the Checks for one of the apps we develop

One good thing with Pingdom is that you can expose these results in public webpages. This means you can share test results for all or some of your URL Checks without requiring a login. An annoying thing with Pingdom is that you have to set up all the Checks by hand. You have to inspect your app code, find the URL, payload,  HTTP method, and enter those manually.

Runscope

Runscope is more tailored towards testing API's, but will also allow you to set up automated response and uptime tests. Below are the results from testing a set of five endpoints used in an app over the course of a day.

Runscope shows you the median response time (50th percentile), as well as the worst-case response (99 percentile) of the endpoints.

Runscope also allows you to do a bit of scripting and parametrization of your queries, which makes it trivial to build test URL's that are truly random.

One nice feature in Runscope is that it will give you very detailed information about request and response payloads for the whole app. It does this by serving as a middle-man between your app and whatever API endpoint it's using. The only thing you need to do as an app developer, is to use a mangled domain-name which allows runscope to receive the request and pass it on to the intended recipient. After that is set up, all requests will appear in your Runscope dashboard, available to be inspected in detail.

Pricing and conclusion

With Pingdom you pay for a limited number of URL Checks while with Runscope you pay for a limited number of HTTP requests. So in short, if you want to test often, but only against a limited set of endpoints, then choose Pingdom. And if you want to test against many endpoints, but not necessarily that often, choose Runscope.

Pingdom has a free tier that checks 1 URL Check. Their cheapest paid tier gives you 10 URL checks and costs around 100USD a year. Runscope has a free tier that gives you 10.000 requests a month, and you can test as many endpoints as you would like. The cheapest paid option from Runscope is roughly 590USD a year with 100.000 requests a month.

Your mileage may vary. We've tried both of the services for a bit and at the moment Runscope seems to fit the needs of an app developer best.