Alibi: grade coding agents on what actually happened on disk
Ask a coding agent whether it finished the job and it will usually say yes. The closing summary is fluent, itemised and confident, and none of it is evidence of anything. We have written before about why logs are not evidence: a record produced by the system it describes inherits that system's honesty. An agent's account of its own run is the same defect one layer up. So we built a harness that ignores the account entirely. Alibi runs the same missions through agentic CLIs and grades every run against ground truth on disk — the files that exist, the changes that were made — never against what the agent says it did. It is open source under MIT: github.com/Traceseal/alibi.
The grading rule
Each configured CLI (currently Claude Code and OpenAI Codex) receives an identical mission brief inside its own bubblewrap sandbox: a private working tree, an isolated home, an isolated /tmp. When the process exits, Alibi inspects the tree. Does the file the mission asked for exist? Does it contain what was required? Did the tests the mission specified actually pass? Did anything change outside the permitted area? The agent's exit message is not consulted at any point. A mission passed if the disk says it passed.
The sandbox does two jobs at once. It keeps the comparison fair, because every lane starts from the same clean state. And it turns containment into something measurable: a write that lands outside the allowed tree is not a style complaint, it is a recorded violation. The full evidence tree — working directory, sandbox home, temp space and logs — is retained after the run, so a grade can be re-derived and argued with after the fact rather than taken on trust.
What Alibi refuses to claim
- It is not a leaderboard. The mission set is small and the runs happened on our machines. A handful of missions is a probe, not a census, and we are not publishing rankings from it.
- It says nothing about consent behaviour. Lanes run with approvals granted by design, because the object of study is execution, not permission-seeking. How a CLI behaves when it has to ask first is a separate question, and this harness does not answer it.
- The grades are Alibi's judgement. Every assertion is ordinary code in the mission definition. If you think a check is unfair, you can read it, re-run it and dispute it — which is more than can be said for a transcript.
The grader cannot be trusted either
A grading harness has the same structural problem as the agents it grades: it produces the record of its own run. If our scores mattered to anyone, we could quietly edit them. As of this week, Alibi closes that gap with its own medicine. Point one environment variable at an Ed25519 key and every (CLI, mission) result is sealed into a Traceseal receipt — hashes of the mission definition, the result record and the entire evidence tree — and a run-level receipt chains the lot into one verifiable unit. Signing happens on the host after the contained processes have exited; the key is never mounted inside the sandbox.
A second party can then act as a witness: re-derive every hash from the retained evidence, re-check every signature and the chain, and cosign the run only if it all matches. The witness tool refuses to cosign with the original signer's key, so a run can carry two independent attestations over the same bytes. One honest caveat travels with the feature: two keys on one machine prove key separation, not operator separation. Full independence requires the witness to verify on hardware the original signer cannot write to.
What the signature means: "this run occurred as described." The scores remain Alibi's judgement; the signature makes them tamper-evident, not true.
What to ask of any agent benchmark
- What record was it graded against? If the answer is the agent's final message, the benchmark measured prose.
- Could the operator rewrite the results afterwards? A results table in a repository the grader controls is a log, with everything that implies.
- Can you re-run it yourself? Missions, assertions and harness should be readable and executable without the author's cooperation.
- What was the consent model? Auto-approved runs measure capability. They do not measure conduct, and results that blur the two flatter everyone.
Alibi is our attempt to be able to answer those four questions cleanly for our own testing. Whether the missions generalise beyond the behaviours we care about is genuinely open — which is one reason the whole thing fits in an afternoon's reading.