Why we gate agent work on evidence
Passing unit tests prove little. Verify the whole feature, the way a user meets it, with the check that fits the use case.
TL;DR
- An agent’s report is text. “All tests pass” is a claim, not proof.
- Passing unit tests prove little. They check the parts, not the feature.
- Verify the whole feature, the way a user meets it. The right check depends on the use case.
- A user journey needs a browser run. An engine needs golden and regression tests. An AI assistant needs a test set.
- Prove each new test can fail. Run the full check on a clean copy. A red check stops the line.
The problem
AI agents write code fast. They also report on their own work, and the report always sounds sure. “Done. All tests pass.”
That report can be wrong in many ways. The tests did not run. They ran, but against old code. They passed, but they would also pass against broken code. Or the agent changed the test to make it pass.
None of this needs bad intent. The agent wants to finish, and a green report looks like finished. So we stopped reading reports as proof.
Passing tests mean little
A green unit test suite says one thing: each part does what its test expects. It does not say the feature works.
There are three reasons.
- The parts can pass while the whole fails. The button works and the API works. But the button calls the wrong API.
- The agent writes the code and the tests together. If it misunderstood the task, the tests carry the same mistake. Both are wrong, and both agree.
- A test can pass against broken code. It checks a mock, or a value it set itself. Nothing real is under test.
So we treat a green unit test run as a start, not as an answer. The question is always: does the feature work for the person who uses it?
Verify the whole journey
A feature is done when its whole journey works, end to end, on the running system. Not before.
The right check depends on the use case:
| Use case | What proves it works |
|---|---|
| A web page or app flow | A browser runs the user's journey: click, type, read the screen, at desktop and phone width |
| A calculation engine or data pipeline | Golden tests with known answers, and a regression run of new against old |
| An AI assistant or agent | A fixed test set of real questions and chats, with a pass rule per group |
| An API or integration | Calls to the running service, checked against recorded real replies |
| A release | The same version passes in the test environment before production |
A user journey: drive the browser
For anything a person clicks, the proof is a browser run. An agent with browser tools opens the page and follows the journey: it clicks, types and reads what the screen shows. Then it does the same at phone width.
This catches what unit tests cannot: a link to a missing page, a button hidden behind another, a layout that scrolls sideways on a phone. On this site, each change is checked this way before it merges.
An engine: golden and regression tests
For a calculation, the proof is the answer. A golden test gives fixed inputs and an answer that a person checked once. The engine must give the same answer every time.
A regression run compares the new version with the old one, case by case. Each difference gets an owner and a reason. Either the old answer was wrong, or the new one is.
An AI assistant: a test set
An AI answer is not the same twice, so an exact match does not work. The proof is a fixed set of real questions, each with a short note on what a pass looks like. The KPI assistant test set shows how one is built.
What counts as evidence
Evidence is something a person can check without trusting the agent.
| The claim | The evidence |
|---|---|
| Tests pass | The check ran on a clean copy, and its output is attached |
| The fix works | A test failed before the fix and passes after it |
| Nothing else changed | The diff touches only the files the ticket names |
| The test is good | The test fails when the code is broken on purpose |
The last row matters most. A test that cannot fail proves nothing. It only adds a green tick.
The gates
Each gate checks one thing, and runs whether or not the agent asks for it.
- 1Commit hooksEvery commit, every agent
- 2Clean-copy checkThe full check, from scratch
- 3Mutation checkEach new test must fail on broken code
- 4Journey checkThe whole feature, as a user meets it
- 5Regression and release gateWhat worked still works
- 6Merge on greenThe gates decide, not the report
Commit hooks
Git runs hooks on every commit, for any agent and any person. They block a commit with no ticket, a commit on the main branch, or a commit holding a secret. Read one local setup for every coding agent for the details.
The full check on a clean copy
An agent’s workspace can hide problems. A file it forgot to add still exists there. A cached build still holds old output.
So the full check runs on a clean copy of the commit. The copy installs its packages from scratch, builds, and runs every test. If it passes there, it passes for the next person too.
A mutation check for each new test
When an agent adds a test, we break the code that the test protects. The test must fail. Then we put the code back.
We report the result as a count: caught out of total. “Four of four” is evidence. “I added tests” is not.
A journey check for each feature
The journey check from the table above runs before every merge. The agent attaches what it saw: the screens, the answers or the test set results.
Regression suites and a release gate
New work must not break old work. For the metric engine, regression and golden suites run against every release.
A release gate sits on top. A release cannot reach production until the same version has passed in the test environment. The KPI assistant has its own fixed test set, and it ships only when that set passes.
Merge on green: a choice of trust
When every gate is green, the work can merge. The decision sits in the gates, not in the agent’s report.
Who merges is a choice. It depends on how much you trust the setup:
| The setup | Who merges |
|---|---|
| New, with few gates | A person reviews and merges every change |
| Gates in place, journey checks passing | An agent merges on green. A person spot-checks |
| Journey checks proven over time | Agents merge routine work. People own the gates |
The condition for an agent merge is simple: the end-to-end journey checks pass. Unit tests alone never earn it.
This is where we are heading. Agents merge, and people spend their time on the gates and on the changes that need judgement. On this site, agents already merge on green, after the clean-copy check and a browser check.
People own the gates either way. A person decides what each gate checks, and changes a gate when it misses something.
When a gate goes red
A red gate stops the line. Nobody starts new work until it is green again.
A flaky check gets the same rule. A test that fails one run in ten teaches everyone to ignore red. Fix it, or remove it and say why.
What a gate does not prove
A gate proves only what it checks. A full green run says the checks passed. It does not say the checks cover the right things.
So a person still reads the change. The gates take away the routine checks, so the review can focus on what matters: is this the right change?
What I learned
- Do not read the report. Run the check. A claim costs nothing to write.
- Unit tests are not the proof. They check the parts. Verify the whole journey.
- Fit the check to the use case. A browser for a journey, golden tests for an engine, a test set for an assistant.
- A test must be able to fail. Prove it once, when you add it.
- Check on a clean copy. The agent’s workspace is not the world.
- Keep the gates outside the agent. A gate the agent can skip is advice, not a control.
- Let trust decide who merges. When the journey checks pass and keep passing, agents can merge.
For the team-wide version of these rules, read how to set up AI governance for any repo.