Test Automation Roadmap: Enterprise 🌍
The Test Automation Roadmap is a model that splits the complex topic of test automation into five mindsets. This article covers the Enterprise mindset. This article is part of a bigger story on the Test Automation Roadmap. You should start with the introduction.
Compliance at scale
You automate a small but crucial part of quality control for multiple departments or the entire (global) enterprise organization. With these tests, you work on compliance at scale. You generally focus on security compliance, law compliance, and self-imposed compliance. Self-imposed compliance can include anything important to your organization.
The Enterprise mindset is very different compared to the others.
Important
Every test in this mindset has to run centrally without the teams to which the tests apply having to do anything. The teams are already busy making and maintaining tests in the other mindsets.
Your tests can be slow as long as they are invisible and non-blocking. You never want to block a release pipeline because a central test is still running or because it failed. If the team needs to change something because your enterprise-level test failed, it should be arranged via management and not enforced by code.
However, if your tests are fast (under ~10 seconds), they can be visible and blocking. However, blocking releases like this can get really tricky really fast. When blocking tests, always build a way around the block through configuration, exceptions, waivers, etc. Otherwise, you will run into issues with legacy applications, third-party applications, and overworked teams.
A big part of making tests in this mindset is sharing your knowledge with teams they apply to. You tell everyone why the tests exist, how they work, and how to interpret the results. You also tell everyone about changes. Especially if these changes can cause a team to no longer be compliant, becoming non-compliant should never surprise teams.
Problems
The technical side of tests in the enterprise mindset is generally very complex. You will need significant programming knowledge to make these tests. Since most testers don’t have this much programming knowledge, developers write the tests instead.
No matter the technical complexity, the human side is more complex. How do you get all teams to comply with your tests? How do you get the teams to fix issues in a timely manner?
Because of the technical and human complexities, you need a dedicated team to make the tests and manage everything around them. A dedicated team requires a significant investment from the organization. However, it will not work without this investment. You can’t adopt this mindset effectively without focus and you can’t achieve anything without mandate.
Tools & Techniques
Risk-based testing is the place to start due to the monetary- and time investments involved. You don’t want to spend a lot on tests that add little value. Once you’ve identified what you want to test, communicate clearly, loudly, and often to get as many teams on board as possible.
Communication with the teams under test is crucial. The need for communication will never go away.
Some general advice:
- Keep any test you write as simple as possible. It will help with both the technical side and communication.
- Static code analysis is a great way of ensuring code compliance. Source code repositories are a goldmine of data ready to be used for security compliance checks. The performance of this approach also scales well.
- Data analysis is a great way of detecting (test-) data compliance issues. Crawling databases can help with law compliance (e.g. GDPR). It can detect incomplete data, duplicate data, and much more.
- Stick with static analysis of some sort. Any other type of testing does not scale well enough to be used in this mindset.
- Use periodically scheduled pipelines by default. When you use release-blocking tests, you can use continuous integration pipelines or build your tests into an internal application (e.g. API gateway).
- When you use release-blocking tests, allow the teams to circumvent (part of) your tests without your intervention. At the same time, the easiest way to do a thing should be the compliant way. Intentionally add speedbumps when teams circumvent your tests but don’t throw up roadblocks.
Position in the model
In the Enterprise mindset, you’ll automate only a few tests. But you’ll do so on a large scale. You focus on adding maximal value to as many teams as possible while you keep the technical and communication complexity minimal. The only way to do this is by strictly scoping your testing efforts. The curved right border on the Enterprise mindset shows us that we run fewer tests if we include more departments.
Real-world example
A short story follows about how a team used the Enterprise mindset in a company of about 45000 employees (according to LinkedIn). The integration department was responsible for providing the platforms that connect all applications in the organization.
The department wanted every new API to be design-first. First, the team writes an OpenAPI Schema (fka Swagger). Based on the OpenAPI Schema, they implement the API. Because of this, every API should be documented by an OpenAPI Schema. In the real world, many teams ignored this way of working. This resulted in both badly designed and undocumented APIs.
At the same time, the API gateway used by the organization was made end-of-life by the vendor. When looking for a new API gateway to migrate to, the department looked for ways to reject new APIs based on the results of tests. This way, they could require all migrated APIs to have an OpenAPI Schema. However, they did not enforce schema completeness. Because of this, some schemas are incomplete.
The next step for this department is to make these tests stricter. This way, they can slowly increase the quality of the OpenAPI Schemas and thus the quality of their API documentation.
Conclusion
The Enterprise mindset is about testing compliance at scale. These tests require a lot of technical complexity, but the human complexity is even greater. Because of this, you need a dedicated team with an explicit mandate that works with a clear scope. This team makes the tests and communicates about them to the organization.
The easiest way to do a thing should be the compliant way. A team can be non-compliant, but it should never be easier than doing the work to become compliant.
