Evaluations

How evals work

An eval runs the same tasks with your skill and without it, in fresh sandboxes on your machine, and tells you whether the skill made the agent better. Every number it reports comes from comparing two runs on the same input.

The question

An eval answers one question: is an agent better at a task with this skill loaded than without it, and better than the version your team already has?

npx -y terum-skills@latest eval <skill>

The skill is a folder in your Library: Global or a project you registered. A team is optional. Without one, the eval still runs; with one, it adds a third comparison and a place to share the result.

Arms

Each task runs once per arm, and each arm gets its own fresh sandbox directory.

ArmWhat is stagedWhen it runs
BaselineNothing. The agent works without the skill.Always
CandidateThe folder on your diskAlways
IncumbentThe team’s published version with the most recent receiptWhen your team has an evaluated version whose bytes differ from yours

Every arm is a real headless Claude Code session on your own login, with the same tools and the same note that no human can answer questions. The skill is staged without its own evals/ folder, so it never sees its answer key. After each session the engine reads which skills the session loaded, and refuses the whole run if the skill leaked into an arm that should not have it.

The same task drops into each sandbox; only the candidate and incumbent have a skill staged, and each session ends in pass or fail on the case’s checks.

What happens in a run

  1. Hygiene. The folder goes through eight deterministic hygiene checks, free and offline. An error stops the run before anything is billed.
  2. Preflight. One tiny real task confirms claude is installed and logged in.
  3. A test set. Cases come from the folder’s evals/. When it has none, the run generates them and writes them into the folder for you to review.
  4. The trigger eval. Each trigger prompt goes to a model with your Library’s skill catalog, which says whether it would pick your skill. No sandbox, no agent.
  5. The arms. Each case runs in every arm, in its own sandbox.
  6. The report and a receipt. The verdict prints first, and a receipt is written on your machine.

Checks decide, the judge breaks ties

Each case, repetition and opponent produces one comparison row. A row is decided by the case’s checks first: deterministic assertions like “this file exists” or “the transcript names applyDiscount”. Checks are all-or-nothing per arm, so an arm passes a case only when every check passes. If one arm passed and the other did not, that is a win or a loss.

When both arms land the same way and the case carries a judge rubric, a model compares the two transcripts. It is asked twice, with the transcripts in both orders. If the two answers agree, that is the result. If they disagree, the row is a tie: a verdict that flips with the order is thrown away rather than kept as a coin flip. A case with no rubric ties.

An arm that crashes or times out is an unscored hole, never a zero. It is left out of every number and shows up as a partial run.

Net lift and the verdict

Net lift is wins minus losses, divided by all scored rows: (W − L) / (W + L + T). Ties stay in the denominator, so they dilute: five wins and five ties is +50%, not +100%.

Rows arrive as wins, losses and ties; net lift moves with them, and the verdict is read from where it lands against the plus and minus one third lines.

The verdict bands the candidate against the baseline:

Net liftVerdict
+1/3 or morePASS
BetweenNEUTRAL
−1/3 or lessFAIL

The dead zone in the middle is deliberate. A result that could be noise is announced as NEUTRAL, not as a small number. The report also prints a sign test for each comparison, and a cost line per arm (turns, time and dollars, as Claude Code reported them), so a skill that wins but triples the bill says so.

Cost and confidence

A run is cases × k × arms full agent sessions, and every one bills your own Claude account. There is no API key and no spend cap. The defaults are one repetition per case (--k 1) on sonnet.

  • --triggers-only runs the trigger eval alone. It is the cheapest useful signal.
  • --case <name> runs one authored case.
  • --k 3 is the setting for a receipt you intend to gate on. At k of 1, a verdict near a band edge is one repetition from flipping.

Where the result goes

The receipt and the full run (transcripts, sandboxes) stay on your machine, filed under the digest of the exact bytes that were evaluated. The receipt reaches your team only when those bytes are already a published version; otherwise publish attaches it later. Results and receipts covers what each number means.

Publishing also reads your receipts: if the newest eval of the bytes you are about to publish is FAIL, publish asks before it goes ahead.

Arms are plain temporary directories, not containers. A skill under test runs on your machine with your permissions, so hygiene and your own review of a teammate’s skill are the first line of defense.

Running evalsEvery stage of a run, the options, batches and the overnight queue.Writing good evalsCases and triggers that measure the job, not the skill’s vocabulary.Test cases and triggersThe file formats and the seven check kinds.Results and receiptsReading the report, the receipt, and how scores reach cards.

Full reference on GitHub