Your suite passed. Your changed lines never ran.

A passing test suite and an untested change are not opposites — they routinely coexist. The suite goes green because the tests you already had still pass. Whether anything executed the lines that were just added is a separate question, and on most teams nobody is asking it.

Here is the real output that makes this concrete. A small billing module: subtotal exists and is tested. An agent adds applyDiscount and no test for it. Then:

  RELAY · trying this repo

  runner    npm test   ·   coverage  node --test (built-in coverage)

  ran your tests once                                    191ms

┌──────────────────────────────────────────────────────────────┐
│   3                                                          │
│   of your 4 changed lines are executed by no test            │
│                                                              │
│   rates.js         4 changed     1 executed                  │
└──────────────────────────────────────────────────────────────┘

  Your suite passed. 3 of your 4 changed lines never ran.

Both sentences in that last line are true at once. That is the entire problem.

Why does this happen more with AI-generated code?

Three reasons, and none of them is "the model is bad."

• Volume. An agent produces working-looking code faster than anyone reviews it. The bottleneck moved from writing to verifying, and most teams have not moved their process.
• The agent grades its own homework. When the same loop writes the code and the test, the test tends to exercise what the agent thinks it built — not what it actually changed. A test can pass while calling none of the new lines.
• Green is convincing. A passing suite looks like evidence. It is evidence of exactly one thing: what was already covered is still covered.

What does "changed-line coverage" measure?

Not project coverage. Project coverage answers "what fraction of the codebase runs under test" — a number that can be 80% while the diff in front of you is 0% executed. Changed-line coverage takes the diff, takes the coverage report of one real test run, and intersects them: which of the lines this change touched were executed by that run?

That intersection is a fact, not a judgement. It is either measured or it is not, and when it is measured, "done" stops being something a model asserts and starts being something a coverage report shows.

What do you do about it?

Run one command in the repository, before anything is merged:

relayevals try

It runs your tests once with coverage, intersects the result with your diff, and shows the panel above. No setup, no account, and it writes nothing — no config, no files in your repo. If the answer is "every changed line ran," you learned that in one command. If the answer is what you see above, you learned that too — before it merged, not after it paged you.

The version of this that gates a merge — signed evidence, a deterministic verdict, an agent loop that drives itself to PASS — is what Relay is. But the habit starts smaller: stop treating a green suite as evidence about lines it never executed.