Contract-based testing 4: Custom build, just for you
In part 3, I’ve started explaining how contract-based testing works. To that end, I’ve talked about the provider-driven approach.
In this part, I want to add to that by talking about the consumer-driven approach. There is more to unpack here than in the provider-driven approach. Because of this, the comparison between the two approaches will have to wait until part 5.
Consumer-driven contract-based testing
In the consumer-driven approach, the consumer writes the contract. Through the contract, the consumer dictates the functionality of the interface. The consumer makes sure that their application can fully handle the interface, as described by the contract. When they’re sure about the contract they share it with the provider. The provider uses all consumer contracts to test their interface.
When there is more than one consumer, they will each write their own contract. This makes the approach great for finely tuned interfaces with one or two consumers.
The contract
A consumer-driven contract contains request-response pairs for the interface. These request-response pairs take the form of ‘If I request A, the interface must respond with B’. This allows the contract to define what the interface should look like and how it should behave. Only the consumer can change the contract.
Request-response pairs will sound familiar if you’ve ever written stubs, as this is how most stub frameworks work. This allows the consumer to use their contract as a source of stubs. At the same time, the provider will use the contract as a source of tests.
Consumer-driven contracts are example-based. Because of this, they’re not great human documentation by themselves. However, add some tooling in the mix and you’ll get great documentation full of examples.
A high-quality consumer-driven contract …
- Contains all requirements the consumer has of the interface.
- Contains as little as possible. Only include the required parts of the interface. Omit everything your application does not need. This allows the provider to make data-driven decisions about their interface.
- Adopts all standards and guidelines. Think of the contract standard itself, company standards, and general best practices.
Where to start
Working contract-first allows us to use the contract as the interface’s specifications. The consumer starts by defining all requirements in the contract. After that, the provider starts implementing the interface. This is a huge process advantage for both sides. It removes a lot of uncertainties, assumptions, and miscommunications throughout the development process.
When working implementation-first, the consumer-driven approach is not the right choice. It does not offer big advantages over the provider-driven approach, but it does take significantly more effort on the provider-side. However, things are never that simple. There are some situations where implementation-first can work. Proceed with caution.
Consumer responsibilities
The consumer has significant power over the interface at every stage. They can both impact the interface and the provider’s workflow. All power comes with responsibilities, the consumer’s responsibilities include:
- Writing the contract
Part of this is ensuring that the contract is of high quality. A high-quality contract directly leads to high-quality contract tests and a lean interface.
When writing the contract, it’s important to be aware of the impact it might have on the provider. A small change on the consumer-side can cause havoc on the provider-side. - Connecting to the interface
This makes you a consumer in the first place. - Testing the consuming application with the contract
The consumer ensures their application can handle everything defined in the contract.
This allows the provider to safely assume that everything the consumer needs the interface to do, is correctly defined in the contract. Skipping this step can result in the provider implementing the wrong thing.
It also makes the contract an integral part of the application’s tests. This coupling ensures that the consumer updates their contract when their requirements change. - Publishing the contract
The final responsibility of the consumer is to publish the tested, high-quality contract to the central contract repository.
Provider responsibilities
In the consumer-driven approach, the provider starts with an assumption: The contract contains everything the consumer needs the interface to do. Nothing more, and nothing less. The provider builds an interface that works with all contracts. Their responsibilities include:
- Downloading all latest contracts
Every test-run the provider downloads the latest contract of every consumer from the central repository. This ensures that the provider is never testing with a missing, or outdated contract. - Implement based on the contracts
The provider builds its interface so that it can do everything described in the contracts. The provider should yell “YAGNI! KISS!” when adding things that are not described in any contract. This keeps the interface free of bloat. - Test implementation with the contracts
The provider uses the request-response pairs in the contracts for black-box testing. This ensures that the interface implementation conforms to the consumer’s requirements. When a test fails the provider must assume that it will cause a production issue on the consumer-side. A failing test must, therefore, always stop a release. - Manage contract test dependencies
The provider has the responsibility to make all tests pass. This includes maintaining test dependencies like data, files, stubs, other contracts, etc. Because of this, every contract increases the test load on the provider. This makes it hard to scale the number of consumers.
An interesting effect of this approach is that the provider gets a lot of feedback from the consumers via their contracts. When the provider removes a part of the interface, the contract tests will reveal exactly who is using it. For example, the provider can find out who is still using that deprecated part of the interface. After all those emails! The contracts contain valuable usage data. The provider can use it to make data-driven decisions when working on the interface.
Another effect worth pointing out is that the provider has to write fewer tests in this approach. Technical and some functional tests are written for them by their consumers. However, this does not mean that the provider doesn’t have to test at all. They have to supplement the contract’s black-box tests with tests of their own.
Summing up
In the consumer-driven approach, everything starts with the consumer. Every consumer is responsible for writing a lean, high-quality contract. The contract dictates the functionality of the interface. The consumers share the contract with the provider via the central contract repository. The provider downloads all the latest contracts every test run. They use the contracts to implement and test their interface.
The good stuff
The consumer-driven approach is a good way to ensure technical and some functional correctness on both sides. The number of functional things that can be tested with this approach depends on the application.
With consumer-driven tests, everything runs locally. This allows for fast test execution on both sides. Test creation is fast on the consumer-side because they have full control over all parts of the tests. On the provider side, it’s a little slower because the consumers write the tests. The provider has no control over them (more on this later).
The consumer-driven approach is great on a process level. Consumers have a lot of control over the interface. It allows them to very clearly and very precisely define what they need the interface to do. They define this in the common language that is the contract standard. For every one that speaks the common language, it’s clear what the interface has to be capable of.
From the provider’s point of view, the consumers define the interface with tests. This gives the provider the process advantages that come with test-driven development (TDD), though without its fast cyclic nature. The contract tests contain usage data for every consumer. By using this data, the provider can make data-driven decisions about the interface.
The bad stuff
The consumers have a lot of control over the interface in this approach. Which is generally a good thing for consumers, but it can be a bad thing for the provider. The provider has full control over all test dependencies, but the consumers control the tests themselves. This creates a dependency between the provider and every one of their consumers. Most problems caused by this can only be solved by communicating with each other. This can impact the provider’s autonomy by tightening the dependency on their consumers.
The most notable example of this is when a consumer publishes a bad contract. A simple typo from a consumer can cause a failing test on the provider-side. It renders the provider incapable of releasing anything to production. This includes releasing security patches and bugfixes. The provider has to wait on the consumer to fix their contract before they can get on with their release. At Pact, they’ve written an article on the issue.
In the consumer-driven approach, it can be difficult to scale the number of consumers. The provider has to ensure that all contract tests pass. Because of this, the test load on the provider increases with every consumer. The test load for a single consumer is very manageable, but test dependencies for 10 consumers is a whole different beast. Besides, having multiple consumers independently write tests like this can result in …
- Redundant tests
Some things will be tested multiple times. Though, in practice, the effect of this should be minimal due to fast test execution. - Contradicting tests
Multiple tests with the same input, but a different output can make it impossible for all tests to pass. Again, communication is the only way to solve this.
Conclusion
In the consumer-driven approach, the consumers define the interface through their contracts. The consumer is responsible for writing and sharing a tested, high-quality contract. The provider downloads all the latest contracts every test run. They use the contracts as a source of tests for their interface.
This approach ensures technical correctness as well as some functional correctness. Working this way is great for building finely tuned interfaces without any bloat. It’s not great for scaling the number of consumers. Some issues in this approach can only be solved by the provider communicating with their consumers. This makes the approach not suitable for internet-facing interfaces.
The consumer-driven approach is a powerful way of working for building an interface with a few consumers. When using it you should be aware of its shortcomings and inherent downsides.
Now that we’ve looked at both approaches, we can start comparing them. In the next part, I’ll compare the provider-driven and consumer-driven approach.
