The blind agent — a driver that read no documentation and reached PASS

The acceptance test for Relay 0.6.22 was a deliberately dumb driver. It had never seen our documentation, our website, or the repository it was dropped into. It was allowed exactly four things: read relayevals agent --format json once, run relayevals next --format json, execute the returned action verbatim, and branch on the exit code. It reached a measured PASS in three turns.

That test exists because of a rule we set ourselves: if the dumb driver stalls, the product gets fixed — not the driver. "Time to green" must not depend on the agent being clever.

Why make the binary the documentation?

Because an agent following a stale document is exactly the failure a verification tool cannot afford. Any hand-maintained inventory of commands and codes drifts from the binary the day after it is written. So the contract is generated, not written: relayevals agent --format json walks the actual command tree and the actual exit-code and remediation tables, and emits all of it — 61 commands, 14 states, 43 finding codes with what clears each — plus the commands already resolved for the repository it runs in:

"repository": {
  "task": "add a subtract function",
  "test_command": "npm test",
  "evidence_command": "relayevals guard \"add a subtract function\" --
    node --test --experimental-test-coverage --test-reporter=lcov ...",
  "coverage": "node --test (built-in coverage)"
}

The flags are pre-resolved for that repo's stack. The agent never guesses.

What did the blind agent actually do?

Two repositories, no coverage tooling in either, one untested change in each. The transcript, verbatim from the run:

turn 1: UNRESOLVED/EVIDENCE_MISSING (exit 20)
turn 2: BLOCK/SURFACE_UNTESTED     (exit 10)
turn 3: PASS                       (exit 0)

Turn 1: no evidence exists yet — the driver runs the evidence command it was handed. Turn 2: evidence now exists and it is bad news — the coverage report proves the changed lines never ran, and the action names the file that needs a test. Turn 3: with the test written, every changed line has measured evidence, and the loop ends.

The one thing the script could not do blindly was write the test itself — that is authorship, the one contribution a language model actually makes in this loop. Everything around it is mechanical, and that is the point: the loop requires no judgement, so a mediocre agent and an excellent one reach the same verdict.

What did the blind agent find?

Real defects, immediately. action.description was prose written for humans — a driver executing it verbatim got Run:: command not found. So actions now carry action.command: the command alone, nothing around it. The first run also exposed a case where a missing coverage directory made a user's passing suite look like a failure. Both were product bugs no unit test had caught, surfaced by the simple discipline of letting a script that cannot improvise try to use the product.

If your tool claims agents can drive it, this is a test worth stealing: take away the documentation, take away the cleverness, and see who stalls first — the agent, or you. The full contract is one command away: relayevals agent, documented here.