Contract-based testing 3: Accept, or get out
In part 1, I’ve talked about what contract-based testing is. In part 2, I continued with why we should prefer contract-based testing over end-to-end testing.
In this part, I want to start with how contract-based testing works. While doing so, I’ll also go into the details of the provider-driven approach. Sadly, the consumer-driven approach will have to wait until part 4.
If you remember from part 2: When contract-based testing, we couple each application with a single file: A contract. The contract is the only coupling between applications. We will then use the contract when testing our application in isolation.
Central contract repository
Contract-based testing only works if both the provider-side and consumer-side use the same contract. This means that sharing contracts has to be done right.
To effectively share contracts, we use a central contract repository. This is a place where everyone can discover, publish, and download contracts. The central repository is essential in making sure that everyone is always using the latest contracts. To this end, you must never share contracts via mail, chat, or any other medium that is not the central repository.
The central contract repository is the only infrastructure an organization needs to test contract-based, regardless of scale.
Where to start
Contract-based testing calls for two things to be made for every integration; the integration and its contract. When we start with writing the contract, we creatively call it contract-first. If we start with building the integration instead, it’s called implementation-first.
When testing contract-based it’s always better to work contract-first. A contract is a way to encode specifications. Starting with the contract allows us to use the contract as the specification document during the development cycle. While the interface is being built, we can use the computer-readable nature of contracts to create the tests for our interface. Making the contract a fundamental part of our tests helps us to deliver quality fast.
For existing interfaces, you are working implementation-first by definition. Working this way negates a bunch of the process advantages you get from working contract-first, but you can still test contract-based. How we handle implementation-first depends on your approach to the contract.
Approaches
In part 1, I mentioned that there are two ways to approach the contract; provider-driven and consumer-driven. It’s finally time to take a closer look at these approaches to contract-based testing.
Provider-driven contract-based testing
In the provider-driven approach, the provider writes the contract. The provider makes sure that the contract matches the implementation. When they’re sure about the contract, they’ll share the contract with all consumers. The consumers use the contract as a part of their tests.
When there are multiple consumers they will all use the same provider contract. This makes the approach great for interfaces with multiple consumers.
The contract
A provider-driven contract contains the definition and documentation of the interface. Only the provider can change it. It describes all the technical details like endpoints, HTTP status codes, response body structure, etc.
Technical details are great for computers, but they won’t be the only ones using the contract. To make sure those pesky humans can also use it effectively, the contract contains examples and descriptions. This makes writing a provider contract feel like writing the documentation of the interface, just in a standardized format.
A high-quality provider-driven contract …
- Covers the entire interface. It omits nothing.
- Is as specific as possible so it leaves no room for assumptions. Note that is this goes for both humans and computers. Humans, in particular, are notorious for making wrong assumptions. But computers are also known to do this through defaults.
- Adopts all standards and guidelines. Think of the contract standard itself, additional company standards, and general best practices.
Where to start
Working contract-first allows us to use the contract as a fundamental part of our interfaces tests from the get-go. This allows the provider to catch any misalignments between the interface and the contract throughout the interfaces development cycle. Any misalignment will cause tests to fail on the provider-side.
For existing interfaces, you would generally be best off by generating the contract from the implementation code. This allows the consumer to reap most benefits of the contract, while the provider barely has to change a thing. However, generating a contract comes at the cost of a lower quality contract, especially for human readers.
Both approaches described here ensure that the implementation and the contract are aligned at all times. No matter where you start, you must always make sure that the interface implementation and the contract are aligned with each other. This is essential as it allows consumers to safely assume that the contract describes the truth.
Provider responsibilities
In the provider-driven approach, the provider has full control over all aspects of the interface at every stage. “With great power comes great responsibility”. The provider’s responsibilities include:
- Writing the contract
Part of this is ensuring that the contract is of high quality. This should not be glossed over as a high-quality contract directly leads to high-quality contract tests for both the provider and the consumers. - Building the interface
This makes you a provider in the first place. - Aligning the contract with the interface
Alignment between the interface implementation and the contract that describes it is essential. It allows consumers to safely assume that the contract describes the reality of the implementation. Omitting alignment will cause the consumer to test with bad stubs, which will lead to production issues. - Publishing the contract
The final responsibility of the provider is to publish the aligned, high-quality contract to the central contract repository.
Consumer responsibilities
In the provider-driven approach, the consumer starts with an assumption: The contract describes the interface correctly. The consumer is responsible for testing and implementing the integration on their end based on the contract. Their responsibilities include:
- Downloading the latest contract
Every test-run the consumer downloads the latest contract from the central repository. This makes sure that the consumer is never testing with an outdated contract. - Consuming as described in the contract
When consuming the consumer must never deviate from the contract. Anything that is not described in the contract is subject to change at any given time without notice upfront. When encountering undocumented behaviour, it’s fair to ask the provider to include it in their contract. An incomplete contract is a quality issue for which the provider is responsible. - Creating all stubs
The consumer writes tests to make sure their application works as expected. These tests include stubs for the interface. This allows the tests to be executed in an isolated environment. The consumer bases these stubs on the contract. Note that the provider is not responsible for providing any stubs. - Validating stubs against the contract
Validation ensures a certain quality level in the stubs and will prevent the stubs from drifting out of sync with production. After the stubs are validated, we can use them in the consumer’s tests like normal stubs are. Doing this will catch most consumer-side issues before they make it to production.
Summing up
In the provider-driven approach, everything starts with the provider. The provider is responsible for writing an aligned, high-quality contract. They share the contract with their consumers via the central contract repository. The consumers download the latest contract every test run and use it to validate their stubs.
The good stuff
The provider-driven approach is a great way to ensure technical correctness and alignment on both sides. This might sound very limited, but practice learns that most tests fall in this category. In these tests, we have full control of all parts, including (stubbed) dependencies. We don’t have to wait on other teams and we can set up our tests without a dedicated environment. Doing everything locally like this allows for fast test creation and execution.
Scaling the number of consumers is easy in this approach. All consumers use the same contract and the provider has full control over it and the interface it describes. Any communication about the interface is handled through the contract. All of this makes the approach ideal for interfaces with multiple consumers. It also means that a new consumer does not have to communicate with the provider, they can do everything themselves based on the contract. This makes the provider-driven approach amazing for internet-facing interfaces.
The bad stuff
We can’t do functional tests between applications because data will never leave the isolated test environments on either side.
The provider has full control over the interface which can be bothersome for consumers. Especially because, most of the time, the provider doesn’t know what the consumers require the interface to do. The provider-driven approach is a one-way communication from the provider to the consumer without a way for the consumer to voice their requirements. The provider may choose to listen to their consumers, but they may also ignore them. This makes a provider-driven contract feel a lot like terms and conditions: Accept our contract, or get out.
Conclusion
Regardless of your approach to contract-based testing, you should work contract-first whenever possible. When sharing contracts make sure you always do so via the central contract repository.
In the provider-driven approach, the provider has full control over the interface at every stage. The provider is responsible for writing and sharing an aligned, high-quality contract. The consumers download the latest contract every test run and use it to validate their stubs.
This approach ensures technical correctness and alignment on both sides. It scales well with the number of consumers making it great for interfaces with multiple consumers, and even better for internet-facing interfaces.
In the next part, I’ll talk about the consumer-driven approach.
