Test Automation

A test suite that exists is not the same as a test suite that runs. Test automation is what turns "we have tests" into "every change is gated by tests, in minutes, with a verdict you trust."

The discipline has its own concerns, separate from writing the tests themselves:

• How long does the suite take, and how do we keep it short?
• Which tests are flaky, and what do we do about them?
• What blocks a merge, what only warns, and what runs nightly?
• How do we report results so failures are diagnosable in 30 seconds, not 30 minutes?
• Where does the data each test needs come from, and how do we keep it cheap?

For the platform mechanics underneath (runners, caching, OIDC, secrets), see CI/CD under infrastructure. This section is about what we do with that platform to make the test suite an asset rather than a tax.

Topics

• Pipeline Shape — Pre-commit, PR, merge queue, nightly, release: deciding what runs when, and why each stage has its own job.
• Parallelization & Sharding — Cutting wall-clock time without losing determinism; splitting the suite, isolating shared state, balancing shards.
• Flake Management — Detecting flakes systematically, quarantining without burying them, and treating flakes as bugs instead of weather.
• Coverage and Gates — What coverage actually measures, where it lies, and which gates are worth blocking a merge on.
• Reporting — Surfacing failures so they're diagnosable in 30 seconds; formats, artifacts, retention, and where the message lands.
• Test Data & Environments — Fixtures, factories, ephemeral environments; how tests get their data, and how the environment stays reproducible.

How These Connect

A practical reading order if you're setting this up from scratch:

1. Pipeline shape first — decide the stages and budgets before anything else.
2. Test data — without isolated data, nothing else works.
3. Parallelization — once data is isolated, you can speed things up safely.
4. Flake management — parallelism surfaces hidden flakes; institute the discipline before the count gets out of hand.
5. Reporting — make failures legible; without this, the whole system is opaque.
6. Coverage and gates — last, and lightly. Coverage is a measurement, not a strategy.

A Note on Scope

This section deliberately doesn't cover:

• What to test, or at what level — see Testing Strategy.
• How to write code that's testable — see Testable Code.
• Framework-specific patterns — see the relevant language or framework section (e.g. front-end/testing).
• CI platform mechanics — see CI/CD.

The split is intentional: writing tests is a craft; running them at scale is an operations problem. Both matter; they have different vocabularies and different failure modes.