Nearly all applications developed today will integrate with an external service of some kind (HTTP, Bluetooth, etc.). These integrations can pose problems for automated testing, exploratory testing, and even demonstrating functionality to a stakeholder.
There are a number of ways a developer can deal with external integrations in tests and for local development. In this post, I’ll discuss a few different options and talk about why I think building a fake API inside your application/service is the right way to go.
My Requirements
What’s important when integrating with an API?
A developer needs to validate that the application is parsing responses correctly, handling errors correctly, dealing with network issues correctly, updating a user interface appropriately, etc. And ideally, a developer would be able to validate all of those things both in automated tests and via manual exploratory testing.
In order to meet those needs, the API should allow a developer to:
- Control state in automated system tests
- Control behavior in automated system tests
- Control state and behavior for exploratory testing and demos
- Provide autonomous (somewhat realistic) behavior for exploratory testing and demos
Option 1: Use the Real Thing
One option when integrating with an external service is to just use the real thing. Regardless of the environment where your application is running (development, automated testing, production), the application could always hit the real external service.
Some services will provide a testing/staging version of their API for development purposes. Depending on how that environment works, you may or may not run into trouble with multiple developers and continuous integration servers running automated tests at the same time, or with multiple people trying to do exploratory testing in parallel.
If the service doesn’t provide a testing/staging version and you only have access to the production API, there could be other concerns outside of the requirements defined above:
- Added cost from large test suites hitting the API over and over again
- Running into rate limits/throttling
- Collisions with test data
Setting those aside for now, let’s look at how well using the real API fits our requirements.
Control state in automated system tests
This will vary quite a bit depending on what the external service does, but in many cases, a developer will be limited by the API. For example, an external service that provides current weather information will not provide API calls for setting the current weather.
This will make it impossible to write automated tests for any particular weather response, as the developer will have no control over the state of the external service. It might also be very difficult (if not impossible) to reset the state of things before each test runs.
Control behavior in automated system tests
It is very unlikely that a developer will have the ability to specify how long a request takes, or to force some kind of unexpected error to occur when using the real external service. It might be possible to trigger a rate limit, but that could then affect other developers, tests, or even production users, so it wouldn’t be a good idea.
Control state and behavior for exploratory testing and demos
The limitations on controlling state and behavior for manual exploratory testing and demos are the same as they would be for running automated tests.
Provide autonomous (somewhat realistic) behavior for exploratory testing and demos
This is where developing against the real service really shines. It doesn’t get more realistic than the real thing. You also get the assurance that if the application is behaving as expected, it’s been integrated correctly with the external service.
Option 2: Internal Fake API
Another option is to develop a fake, stand-in API that runs within the application being developed. Most tech stacks will have at least a couple of libraries that can be used to open up a port and handle HTTP requests just by providing the appropriate handlers. If this is done in the same process as the tests are running, then the tests can easily configure the state of the fake API and manipulate its behavior in whatever way is needed.
Since the API is custom software, it can be made to fit any and all of the desired requirements.
Control state in automated system tests
With an in-memory fake API, it should be possible to configure the internal state of the service in whatever way is needed for a particular test. I’ve usually done this by providing functions that can be called from tests to add data or configure the service’s internal state.
It’s also possible (and very straightforward) to provide a function that resets the internal state of the fake API. This is important because it’s something that needs to be done in between automated tests.
A huge benefit of a custom fake API is that it allows a developer to simulate the real API, not just provide hard-coded responses to predefined inputs. If the real API allows a client to add entries and then later query them, a developer could implement that same functionality, using an in-memory hash/map/dictionary as the database.
This means that tests can focus on the application’s behavior, and not have to spend time making sure that each network request has been appropriately mocked out, or that a canned response has been configured, etc.
Control behavior in automated system tests
With a custom fake API, it’s possible to configure a particular request handler to delay its response, to return a specific error, etc. If the fake API is running in the same process, then functions can be accessed from the tests to manipulate the behavior of the API in whatever way is needed.
Control state and behavior for exploratory testing and demos
Scenarios that require rare circumstances or long periods of time to elapse can be difficult to manually test or demo. With a custom fake API, it’s possible to do things like speed up time or put the system into a specific state that might be difficult to recreate under normal circumstances (e.g. simulate the API rate limiting requests).
Since, for the most part, these are HTTP APIs, it’s trivial to add endpoints that allow for state/behavior to be manipulated from outside of the process if desired. This makes it possible to manually run scripts or use tools like Postman to configure the state/behavior of the fake API by hand, as needed for any specific manual testing scenario.
Provide autonomous (somewhat realistic) behavior for exploratory testing and demos
As described above, the fake API can be implemented so that it simulates the behavior of the real API. Depending on what the external service is providing to the application, simulating its behavior can allow a human to use the application and have a fairly realistic experience.
Other Alternatives
There are, of course, other alternatives to using the real API or writing your own custom fake API. My experience has been that some of these other options can be useful, especially when you want to get something up and running quickly.
Use an externally hosted mock API
Services like mockable.io and { JSON: Placeholder } can be used to quickly stand up a placeholder API.
Mock out API calls using a library
Some languages have libraries available for mocking out network or HTTP calls, and they can be used to specify responses for given requests. Ruby has WebMock, and there’s Nock for JavaScript.
I’ve given the library route a few tries and always found it too limiting–especially for system tests or running a local development server against the API. I don’t have much experience with some of the externally hosted mock API services, but my guess is that you’ll run into similar limitations. They’ll be the fastest way to stand up some canned responses, but they can’t compete with the flexibility of the custom API implementation.
Conclusion
I have found success implementing custom fake APIs to stand in for a real external service. They can be used in automated tests, instrumented to behave in whatever way is required for the test at hand. And they can be used to allow running the app under development without needing to access the real API (either because it doesn’t yet exist, or because it’s not a system where you should do local testing against it).
I don’t want to give the impression that there aren’t any downsides to implementing your own custom fake API. It will certainly take more development time, both upfront and over the long run. And it’s more code that needs to be supported. But I’ve found that the value it provides over the course of an application’s lifetime far outweighs its downsides.
I’ve had a lot of success using wiremock (http://wiremock.org/docs/) to build fake APIs. It has all the recording and playback features that let you get started quickly, but I think it has two key advantages over other solutions like that:
1. It can be run as a standalone application, which means you can basically just run it like another web server dependency and point your app at it, and it doesn’t care what language/framework you’re using.
2. It has an extension API (http://wiremock.org/docs/extending-wiremock/) that allows you to write custom JVM code to provide more complex behavior if needed. We’ve used this a couple times for APIs that have multi step transactions we need to mock out.
Thanks Al! I’ll have to check it out.
I have been using Mocklets(https://mocklets.com) to generate mock APIs. It provides features like dynamic responses, API proxy, and randomized responses which are very useful while testing the app against production network scenarios.
Integration support with tools like Charles Proxy makes it very usable during the testing phase.