Results and receipts

#The printed report

verdict: PASS
why: wins on execution checks; loses on judge calls; 3 ties
candidate-vs-baseline: +44% net lift (5W / 1L / 3T over 9 comparisons, sign test p=0.219)
candidate-vs-incumbent: +11% net lift (2W / 1L / 6T over 9 comparisons, sign test p=1.000)
arm scores: baseline 0.61 · candidate 0.82 · incumbent 0.74
case-runs passed: baseline 2/9 · candidate 5/9 · incumbent 4/9
triggers: recall=1.00 precision=0.83 (tp=5 fn=0 fp=1 tn=5)
  FALSE-FIRE: "rename this variable"
efficiency: baseline 3.0 turns · 22.8s · $0.11 | candidate 5.0 turns · 41.2s · $0.20 | incumbent 4.0 turns · 33.0s · $0.15

• verdict: is PASS, NEUTRAL or FAIL, from the candidate-vs-baseline comparison alone. An incomplete run adds how many rows were scored.
• why: is one sentence built from what decided the baseline rows. No model writes it.
• candidate-vs-… lines give net lift, the win, loss and tie record, and the sign test, one line per opponent.
• arm scores: is each arm’s mean fraction of checks passed.
• case-runs passed: counts the case-runs where an arm passed every check. The app calls this Quality.
• triggers: is the trigger eval’s recall and precision, then one MISS: or FALSE-FIRE: line per prompt that went the wrong way.
• efficiency: is each arm’s mean turns, seconds and dollars, as Claude Code reported them. The engine computes no cost of its own.

#How one row is decided

A row is one case, one repetition, one opponent. The first rule that applies decides it:

1. An arm died: a tie, left unscored.
2. One arm passed every check and the other did not: a win or a loss.
3. Both got the same all-or-nothing result and the case has no judge rubric: a tie.
4. Both got the same result and the case has a rubric: the judge decides.

The judge sees the task, the rubric, and the last 6,000 characters of each transcript as A and B, never which arm is which. It is asked twice, once in each order, and if the answers disagree the row is a tie. An unusable answer is retried and then escalated to opus; any judge failure is a tie, never a win. Suites are never judged.

#The numbers

Net lift is (wins - losses) / (wins + losses + ties) over the scored rows. Ties stay in the denominator, so they dilute: five wins and five ties is +50%, not +100%.

The verdict bands net lift against the baseline: PASS at +1/3 or above, FAIL at -1/3 or below, NEUTRAL in between or when there is no baseline comparison. The comparison is done in integers, so the boundaries are exact.

The sign test asks how often a split this lopsided would happen if both arms were equally good. It ignores ties and gates nothing. At the default k of 1 it is rarely small: three wins from three cases is p=0.250. A run that includes a suite pins it at 1.0 in the receipt, because rows sharing one agent session are not independent. The printed line does not show that pin, so after a suite run, trust the receipt.

Repetitions do not roll up. Each case and repetition is its own row, so --k 3 over three cases is nine rows per opponent.

Partial runs. A receipt records expected_rows and scored_rows, and its status is complete, partial or failed. A case skipped for a missing tool, or dropped before it started, is named in the receipt; a dead arm shows as the difference. An unscored hole is never counted as a zero.

#The receipt

One JSON file per run. The fields that matter most:

• verdict, execution_status, and the expected_rows / scored_rows pair.
• comparisons: each opponent’s win, loss, tie, net_lift and sign_p.
• content_digest: a sha256: over the folder as evaluated, the same function publish uses, with eval assets excluded.
• version: null on a local run, because a local eval happens before the bytes have a version number.
• provenance: who ran it (your handle, or local), the engine and Claude Code versions, the requested model and judge model, k, the cases, each arm’s resolved skill list, and the start time, so you can see whether two receipts are comparable.

A receipt’s free text (the why line, check names, dropped-case details) is scrubbed of credential-shaped strings, such as GitHub tokens, Anthropic keys and private keys, before the receipt is written. Transcripts and run trees stay on your machine, unredacted.

#Where a run lands

~/.terum/skills/evals/local/<content digest>/<run id>/
  receipt.json
  run.jsonl
  transcripts/
  sandboxes/
  generated/

On Windows the root is %USERPROFILE%\.terum\skills\. The store is keyed by the bytes evaluated, not by team or version, so a skill in no team can still be evaluated and a receipt can later be matched to exact bytes. Transcripts and sandboxes are kept for you to read.

#When a receipt goes to the team

The eval commits its own receipt only when the bytes it evaluated are exactly a published version of the skill, and you have a team, a handle, a skill id and no --no-commit. The receipt lands at evals/<skill-id>/v<N>/<run id>.json, stamped with that version, and never overwrites an existing file:

Published this receipt to myteam for Version 3 of deploy-check.

Anything short of an exact byte match shares nothing, which is the normal outcome for a folder you have edited. The run prints a hint instead, and the next publish of those bytes attaches every matching local receipt, whether or not it mints a version. Publish also reads them the other way: if the newest local receipt of the bytes is FAIL, it asks before publishing.

#How a score reaches a card

A card shows one receipt’s candidate-vs-baseline comparison beside its provenance. No surface combines two receipts into one number.

• A Marketplace card shows the newest version that has a usable receipt, and marks the score stale when that is not the latest version.
• A Library card is matched on bytes: it shows a receipt whose content_digest equals the folder’s current digest, from this machine first and the team’s receipts second. Editing a skill changes the digest, so its score blanks, marked stale rather than absent when an earlier digest was scored. Adding or regenerating eval files does not blank it.

A teammate’s receipt reaches you through the team repository when your clone next refreshes, and shows on your card when it matches the bytes you have installed.