Every harness can finish. One can tell you whether it did.

The distinction everything rests on

Authoring harnesses are ranked by benchmarks that hold the answer key

Every harness here except Relay is an authoring harness: instruction in, patch out. They are compared on benchmarks that supply a hidden test suite — which means the benchmark itself is the verifier. In your repository nobody holds the answer key. So the question stops being can it finish and becomes: when it says done, is it?

That question has a number. None of them publish it. This page publishes ours.

1 · The field

As it actually installs

Versions read on one machine on 2026-08-24. Four were installed and run here; the rest were inspected through their published packages and documentation. The marker in the last column says which.

Darwin is absent, not omitted. No shipped CLI harness by that name exists on npm, GitHub, or in any 2026 harness survey. The only Darwin in this space is the Darwin Gödel Machine (arXiv 2505.22954), a research self-improving agent — not something you can install.

2 · Capability

Where the architectures actually differ

3 · The measured head-to-head

Same task, same account, same model — an independent judge

5 archetype repositories (esm-root, cjs-lib, src-layout, multi-module, workspace) × 5 runs per cell × 2 phrasings. The judge is independent of every arm: the function must exist, a test must reference it, and node --test must exit 0. Relay’s own verdict is recorded separately as a self-report and is not the resolution metric — a harness does not get to grade itself here, including this one.

An escape is the number that matters: a run where the arm exited 0 and the judge disagreed. It is the machine equivalent of “done!” when it is not done.

Terse phrasing — the words a person actually types

Every arm wrote correct code. The solo Claude Code arm wrote no test that touched it in 3 of 5 runs and reported success anyway. That is not a model failure — it is a harness with nothing positioned to disagree.

The escapes, named

Explicit phrasing — you have already said what done means

This is the half that does not flatter Relay, and it is the honest headline: told exactly what Relay enforces by default, every arm gets there. Enforcement pays against a careless agent and a terse instruction. Against a careful one, it costs you tokens and buys you a receipt.

4 · Can the verifier be fooled

A gate that passes everything is not a gate

The head-to-head only means something if Relay’s own verdict is hard to fake. The red-first corpus is 40 fix-shaped tasks, half of them carrying vacuous tests — tests that still pass with the change reverted. Ground truth is established independently of the code under test: independent of the code under test: a plain copy, `git checkout HEAD -- src.js`, and `node --test src.test.js` run directly. No worktree, no split logic, no interpretation.

Gate for this corpus, fixed before the run: 40 fix-shaped tasks, 20 vacuous; flags >=18/20 vacuous, 0/20 genuine.

5 · The benchmark that grades itself

HumanEval does not execute 15 lines of its own answers

Every capability score above is about harnesses. This one is about the benchmarks. HumanEval ships its tests with the problem, so the question worth asking is not how many problems an agent solves — it is whether those tests execute the solution the benchmark itself certifies as correct.

Measured across all 164 problems with no agent and no model: the reference answer, HumanEval’s own check(), and coverage over the written body only.

The honest size of it: 1.5% of lines. That is the ceiling on how far a coverage-verified score could diverge from pass@1 here, and it is why we did not spend an agent run on it. On a real repository, where the suite is the team’s own rather than the benchmark’s, the same measurement is the whole point — which is what section 3 measures.

An earlier run of this measurement reported three failing reference solutions. That was our bug, not HumanEval’s: three prompts define a helper beside the target and their check() calls it directly, and we had imported only the entry point. The number above is 0.

6 · The standard benchmark, asked the same question

SWE-bench certifies a fix whose lines its own test never runs

SWE-bench Lite is the benchmark this field actually reports. Each instance ships a real bug, the maintainer’s certified fix, and the tests that certify it. That makes it answerable with the same question section 5 asks HumanEval: when the benchmark says resolved, did its test execute the fix?

The agent slot is filled by the instance’s own gold patch, on purpose. Nothing here is a model score. It measures the ceiling: the best a harness could possibly do is verify a fix already known to be correct.

Both instances reproduced completely: the test patch applied, the target test failed at the base commit, the gold patch applied, the test went green, and 60 PASS_TO_PASS tests stayed green. Both receipts verified. Relay returned PASS on one and BLOCK on the other.

The one that blocked is the finding. On psf__requests-3362 the certified fix changes 12 lines in requests/utils.py, and the benchmark’s own FAIL_TO_PASS test executes 10 of them. 2 lines of the official fix are never run by the test that certifies it. That instance scores resolved on the leaderboard.

Two instances is two instances. An earlier run of this script covered nine and its numbers were discarded, because the harness wrote its own runner into each repository without telling Relay to ignore it, so Relay counted an untracked file as an unreceipted changed surface. Relay was right and the harness was feeding it a lie. The two instances above are the ones re-verified after that fix, so they are the only ones this page will claim.

Not a leaderboard submission, and not comparable to one. NOT a leaderboard submission: the official evaluation builds a Docker image per instance and this does not. The agent slot is filled by the instance's own gold patch, so this measures HARNESS correctness (the ceiling), never model performance.

7 · Limits

What this does not establish

Published in full, because a benchmark without this section is an advertisement.

• 5 runs per cell supports very little. The confidence intervals in the tables above are wide on purpose — at this sample size a 0% escape rate is statistically indistinguishable from roughly 43%. It separates a clean arm from a badly leaking one and nothing finer.
• The corpus is deliberately easy. Single functions in five small repository shapes. It does not resemble a real change that touches several files and mixes a refactor with a fix.
• opencode was not run. not installed / not authenticated on this machine — needs founder credentials. Its row in the field table is a published claim, not a measurement, and it is marked as one.
• prime-agent was not run. not installed / not authenticated on this machine — needs founder credentials. Its row in the field table is a published claim, not a measurement, and it is marked as one.
• That red-first works on real-world diffs. Every case here is one file, one function, one runner (`node --test`), and a clean two-way split between source and test. Real changes touch several files, mix refactors with fixes, and use runners this check cannot yet target.
• That these are the only vacuous shapes. Four are exercised — bare call, trivially-true assertion, assertion on an untouched property, and assert-not-stale as the genuine control. A test that asserts on a value the fix changed INCIDENTALLY would read as genuine here and is not covered.
• The SWE-bench section is 2 instances, not a score. Official evaluation isolates each instance in its own Docker image and this does not, so the number in section 6 is a finding about one certified patch, not a resolution rate and not a position on the leaderboard.
• Relay is not neutral here. We built one of the arms and ran the comparison. The mitigation is that the judge is external to every arm, the artifacts are published, and the result that hurts us most — explicit phrasing erasing our advantage entirely — is printed above in the same size as everything else.

Reproduce it

The artifacts, not just the conclusions

Every number above is read at build time from the JSON the run wrote — no figure on this page was retyped by hand. The judge definition it was scored against, verbatim:

independent: function in module + test references it + node --test green; NEVER a harness self-report

Relay itself installs in one command, and the verification it runs on your repository is the same machinery scored above.

curl -fsSL https://relayevals.com/install.sh | bash

Four-command quickstart · How verification works · What a deterministic verdict is