# Results and receipts Every eval prints a report and writes a receipt. This page explains what each number means, how a single comparison is decided, and when a receipt reaches your team. ## The printed report ```text verdict: PASS why: wins on execution checks; loses on judge calls; 3 ties candidate-vs-baseline: +44% net lift (5W / 1L / 3T over 9 comparisons, sign test p=0.219) candidate-vs-incumbent: +11% net lift (2W / 1L / 6T over 9 comparisons, sign test p=1.000) arm scores: baseline 0.61 · candidate 0.82 · incumbent 0.74 case-runs passed: baseline 2/9 · candidate 5/9 · incumbent 4/9 triggers: recall=1.00 precision=0.83 (tp=5 fn=0 fp=1 tn=5) FALSE-FIRE: "rename this variable" efficiency: baseline 3.0 turns · 22.8s · $0.11 | candidate 5.0 turns · 41.2s · $0.20 | incumbent 4.0 turns · 33.0s · $0.15 ``` - `verdict:` is `PASS`, `NEUTRAL` or `FAIL`, from the candidate-vs-baseline comparison alone. An incomplete run adds how many rows were scored. - `why:` is one sentence built from what decided the baseline rows. No model writes it. - `candidate-vs-…` lines give net lift, the win, loss and tie record, and the sign test, one line per opponent. - `arm scores:` is each arm's mean fraction of checks passed. - `case-runs passed:` counts the case-runs where an arm passed every check. The app calls this Quality. - `triggers:` is the trigger eval's recall and precision, then one `MISS:` or `FALSE-FIRE:` line per prompt that went the wrong way. - `efficiency:` is each arm's mean turns, seconds and dollars, as Claude Code reported them. The engine computes no cost of its own. ## How one row is decided A row is one case, one repetition, one opponent. The first rule that applies decides it: 1. An arm died: a tie, left unscored. 2. One arm passed every check and the other did not: a win or a loss. 3. Both got the same all-or-nothing result and the case has no `judge` rubric: a tie. 4. Both got the same result and the case has a rubric: the judge decides. The judge sees the task, the rubric, and the last 6,000 characters of each transcript as A and B, never which arm is which. It is asked twice, once in each order, and if the answers disagree the row is a tie. An unusable answer is retried and then escalated to `opus`; any judge failure is a tie, never a win. Suites are never judged. ## The numbers **Net lift** is `(wins - losses) / (wins + losses + ties)` over the scored rows. Ties stay in the denominator, so they dilute: five wins and five ties is +50%, not +100%. **The verdict** bands net lift against the baseline: `PASS` at +1/3 or above, `FAIL` at -1/3 or below, `NEUTRAL` in between or when there is no baseline comparison. The comparison is done in integers, so the boundaries are exact. **The sign test** asks how often a split this lopsided would happen if both arms were equally good. It ignores ties and gates nothing. At the default `k` of 1 it is rarely small: three wins from three cases is p=0.250. A run that includes a suite pins it at 1.0 in the receipt, because rows sharing one agent session are not independent. The printed line does not show that pin, so after a suite run, trust the receipt. **Repetitions do not roll up.** Each case and repetition is its own row, so `--k 3` over three cases is nine rows per opponent. **Partial runs.** A receipt records `expected_rows` and `scored_rows`, and its status is `complete`, `partial` or `failed`. A case skipped for a missing tool, or dropped before it started, is named in the receipt; a dead arm shows as the difference. An unscored hole is never counted as a zero. ## The receipt One JSON file per run. The fields that matter most: - `verdict`, `execution_status`, and the `expected_rows` / `scored_rows` pair. - `comparisons`: each opponent's `win`, `loss`, `tie`, `net_lift` and `sign_p`. - `content_digest`: a `sha256:` over the folder as evaluated, the same function publish uses, with eval assets excluded. - `version`: null on a local run, because a local eval happens before the bytes have a version number. - `provenance`: who ran it (your handle, or `local`), the engine and Claude Code versions, the requested model and judge model, `k`, the cases, each arm's resolved skill list, and the start time, so you can see whether two receipts are comparable. A receipt's free text (the `why` line, check names, dropped-case details) is scrubbed of credential-shaped strings, such as GitHub tokens, Anthropic keys and private keys, before the receipt is written. Transcripts and run trees stay on your machine, unredacted. ## Where a run lands ```text ~/.terum/skills/evals/local/// receipt.json run.jsonl transcripts/ sandboxes/ generated/ ``` On Windows the root is `%USERPROFILE%\.terum\skills\`. The store is keyed by the bytes evaluated, not by team or version, so a skill in no team can still be evaluated and a receipt can later be matched to exact bytes. Transcripts and sandboxes are kept for you to read. ## When a receipt goes to the team The eval commits its own receipt only when the bytes it evaluated are exactly a published version of the skill, and you have a team, a handle, a skill id and no `--no-commit`. The receipt lands at `evals//v/.json`, stamped with that version, and never overwrites an existing file: ```text Published this receipt to myteam for Version 3 of deploy-check. ``` ::: figure receipt-share A receipt goes to the team only when the bytes it evaluated are already a published version; otherwise it waits on this machine until publish attaches it. ::: Anything short of an exact byte match shares nothing, which is the normal outcome for a folder you have edited. The run prints a hint instead, and the next `publish` of those bytes attaches every matching local receipt, whether or not it mints a version. Publish also reads them the other way: if the newest local receipt of the bytes is `FAIL`, it asks before publishing. ## How a score reaches a card A card shows one receipt's candidate-vs-baseline comparison beside its provenance. No surface combines two receipts into one number. - **A Marketplace card** shows the newest version that has a usable receipt, and marks the score stale when that is not the latest version. - **A Library card** is matched on bytes: it shows a receipt whose `content_digest` equals the folder's current digest, from this machine first and the team's receipts second. Editing a skill changes the digest, so its score blanks, marked stale rather than absent when an earlier digest was scored. Adding or regenerating eval files does not blank it. A teammate's receipt reaches you through the team repository when your clone next refreshes, and shows on your card when it matches the bytes you have installed.