How to verify AI-generated code before you merge

Verifying AI-generated code means establishing, mechanically, that the lines an agent changed were actually executed by a test — before the change merges. It is different from reviewing the code (a judgement about quality) and different from running the suite (which can pass without touching the new lines). This page explains how to do it, what it proves, and what it deliberately does not.

Why is a passing test suite not verification?

Because a suite is green when the tests you already had still pass. It says nothing about whether anything executed the lines that were just added. These two facts routinely coexist:

  Your suite passed. 3 of your 4 changed lines never ran.

That output is from a real run on a real repository: an agent added a function, the suite passed, and three quarters of the change had never executed anywhere. A team relying on "CI is green" would have merged untested code while feeling verified.

Project-wide coverage numbers do not close this gap. A codebase can hold 80% coverage while the diff in front of you is 0% executed — the 80% was earned by old code. The measurement that matters at merge time is changed-line coverage: the intersection of the diff with the coverage report of one real test run.

Why does AI-generated code make this urgent?

Three structural reasons, none of which is "the model writes bad code":

• Volume moved the bottleneck. Agents produce plausible code faster than humans review it. The scarce resource is no longer writing — it is knowing what the code actually does.
• The agent grades its own homework. When one loop writes both the code and the test, the test tends to exercise what the agent believes it built. A test can pass while calling none of the changed lines — and the agent then reports, sincerely, that the work is done and tested.
• Assertions are free. "All tests pass and the feature works" costs a model nothing to say. Without a mechanical check, that sentence is indistinguishable from the true version of itself.

The industry's answer so far has been AI code review — a second model reading the diff and offering opinions. Opinions are useful. They are also not evidence, which brings us to the distinction this page exists to make.

What is the difference between review and verification?

Review is a judgement; verification is a measurement. A reviewer — human or LLM — asks: is this change well-designed, idiomatic, safe, sensible? A verifier asks a narrower question: did anything execute these lines, and can you prove it? The two are complementary, and neither substitutes for the other:

• Review without verification: an eloquent opinion about code that may never have run.
• Verification without review: proof the code ran, with no judgement about whether it is any good.

A sound merge process wants both — judgement in the review, fact in the gate. What it must not do is gate merges on an opinion, or dress an opinion up as a fact. If a model's output can change the gate's answer, the gate has inherited the reliability of model output.

How does verification actually work?

The mechanism, end to end, as Relay implements it:

1. Measure the change. Take the diff — the exact changed lines, per file. Take one real run of the repository's test suite with coverage enabled. Intersect them. The result is a per-line fact: executed, or not.

2. Make the evidence tamper-evident. Wrap the run in a signed receipt: the command that ran, its exit code, hashes of its output, the coverage report, and a snapshot hash binding all of it to the exact repository tree it describes. Ed25519 signatures make the receipt verifiable by anyone and forgeable by no one — including the agent whose work it grades. If a file changes after the evidence was signed, the snapshot no longer matches and the evidence is stale.

3. Compute the verdict deterministically. Same diff, same evidence, same answer, every time. No model output is ever an input. The verdict names its reason:

"state": "PASS",
"reason": "4 / 4 changed line(s) executed; signed evidence valid;
           required checks passed."

4. Let the agent drive — but never decide. The loop is mechanical enough for any agent: ask relayevals next --format json, do exactly the returned action, repeat. When evidence is missing, the action is the exact command that produces it, flags already resolved for the repository's stack. When a changed line has no test, the action names the file. The one decision an agent can never make is vouching for its own signing key — that approval is human-only and terminal-only, and the loop stops and asks for a person.

What if the repository has no coverage tool?

Then verification uses a ladder, and every rung ends in something runnable:

• The repository's own tool (vitest, jest, c8, nyc, coverage.py) — always wins if present.
• The runtime's built-in coverage — node --test on Node 22 needs nothing installed.
• An instrument the verifier carries for itself — Relay installs its own c8 under ~/.relay, never touching the repository, and disclosing the instrument in the signed receipt: name, version, and the sha256 of the exact binary that ran.
• Failing all of that: one copy-paste install command for the stack at hand.

"We do not have coverage set up" stops being a reason a change goes unverified.

What does a PASS prove — and not prove?

It proves: every changed executable line was executed by a test run that was measured, signed, and bound to this exact diff, and the signatures and snapshot check out.

It does not prove: that the code is correct, that the tests assert the right things, that the design is sound, or that you should merge. A test can execute every line and still assert the wrong thing. This is why an honest verifier never says "safe to merge" — the phrase converts a precise measurement into a blanket blessing it cannot back. The narrowness is the point: you know exactly what was established, and what remains yours to judge.

How do you add this to a real workflow?

Start with the zero-commitment version — one command, no setup, writes nothing:

cd your-repo
relayevals try

It runs your tests once and shows which changed lines no test executed. If that number surprises you, the full workflow is two commands for a human:

relayevals start "add a percentage discount to invoice totals"
relayevals trust approve   # once per machine, ~30 seconds

start sets up the repository, resolves how to measure coverage in your stack, binds the task, and prints a handoff block for your agent. The agent then drives itself: produce evidence, meet the BLOCK naming the untested file, write the missing test, reach PASS. At the merge gate, relayevals verdict recomputes the decision from the receipts and exits with it, so branch protection acts on a fact rather than a feeling. Install:

curl -fsSL https://relayevals.com/install.sh | bash

Frequently asked questions

Can an AI agent fake its way past verification?

Not through anything it says. The verdict consumes only measured inputs — coverage from a real run, signatures, snapshot hashes. The receipts an agent would need to forge require a signing key a human approved in a terminal; the loop's contract explicitly forbids handcrafting receipts, and a forged file without a valid signature simply fails verification.

Does this replace AI code review tools?

No — different question. Review tools (compared here) judge quality; verification establishes execution. Teams that want both should run both. What verification uniquely adds is a fact the reviewer can rely on: the code being reviewed has actually run.

Does verification slow the loop down?

One test run per verdict — the run you were going to want anyway. In our release verification, a documentation-blind agent drove an unverified change from BLOCK to a measured PASS in three turns on a clean machine, including writing the missing test.

What about changes tests cannot cover?

Config files, docs, lockfiles have no executable lines and are handled as what they are — not blocked for lacking coverage they cannot have. Policy decides what counts as a change requiring evidence, and the policy file itself is guarded: edit it without a human accepting the change, and the verdict path refuses until someone does.