# Writing good evals A verdict is only as good as the cases behind it. How to write tasks, checks and triggers that separate a skill doing the job from an agent without it. ## Start from the generated set Run an eval on a skill with no tests and it [writes them for you](https://www.terum.ai/docs/generated-evals), into the skill folder. Only a missing kind is generated, so generate first and then edit: one case file of your own turns case generation off. Read what it wrote before you trust the verdict. The generator sees your whole `SKILL.md`, and it fails in two ways: checks that reward the skill's vocabulary instead of the job, and `should_not_trigger` prompts that are not real near misses. ## Write tasks an agent can finish alone Every arm is told that no human can answer questions. An agent that stops to ask scores as a failure, so everything the task needs has to be in the sandbox or in the task. - Write the request a user would make. A demand like "every report MUST start with the literal line X" reads as prompt injection and measures nothing. - Do not say what to look for. Both arms read the task. - Do not start with `/`. Only an arm with your skill has that command, so the row measures the command. - If the agent might do the job without invoking your skill, name the skill in plain words. ## Checks that measure the job Ask of every check: could an agent without your skill pass it by doing the job well? If not, it measures whether your skill's wording reached the context, not whether the work got done. - **Observe outcomes.** A file that should exist, a command that should not run, an end state `command_succeeds` can verify. Transcript checks read what the agent wrote and passed to tools, not what tools returned. - **Checks are all-or-nothing per arm.** Checks decide a row only when one arm passes every check and the other does not, so two of three against none is a tie. Put independent behaviors in separate cases, and drop any check neither arm can pass. - **Measure precision.** Pair `transcript_mentions` or `command_matching` with `transcript_omits` or `no_command_matching`, so a skill that does too much fails the case. - **Anchor on identifiers:** a function name, a constant, an error code, a `file:line`. Matching ignores case and finds substrings, so a prose anchor misses a paraphrase and a short one passes by luck. ## A good case A case for a release skill, saved as `evals/cases/blocked-release.yaml`: ```yaml task: Tag and push the 4.2 release from this repository when it is ready to ship. files: RELEASE.md: | # 4.2 Blocked: migration 0042_add_invoice_index has not run on staging. setup: git init -q && git add -A && git -c user.name=t -c user.email=t@example.com commit -qm "4.2 candidate" requires: [git] checks: - command_succeeds: 'test -z "$(git tag)"' - no_command_matching: 'git push' - transcript_mentions: "0042_add_invoice_index" bucket: adversarial ``` The task is an ordinary request, and `setup` builds a real repository around the one fact that matters. The checks look at the end state (no tag), the action that must not happen (no push) and the identifier a correct answer cites. `requires` turns a machine without git into a visible skip instead of a tie. The case is `adversarial`: the obliging answer is wrong. An agent without your skill tends to oblige, which is why the generator puts at least one such case in every set. ::: note `setup`, `requires` and `command_succeeds` run through `/bin/sh`, so a case that uses them runs on macOS, Linux or WSL. ::: ## Judges, triggers and suites ### When checks are not enough A `judge` rubric is read only when a case's checks tie. The judge sees the task, the rubric and the end of each transcript, never the sandbox. It is asked twice with the transcripts swapped, and if the answers disagree the row is a tie. Use one where two passing answers can differ in quality, keep checks as the floor, and put the facts the judge needs into the rubric. No arm ever sees it. ### Triggers A trigger prompt tests your skill's name and description against the other skills in its [Library](https://www.terum.ai/docs/library-and-marketplace) root. Phrase `should_trigger` prompts several ways: naming the job, describing a symptom, adding context. Draw `should_not_trigger` from the requests owned by the skills yours competes with. `misses` finds prompts you actually typed where a skill should have fired and did not. ### Suites A skill that launches subagents belongs in a [suite](https://www.terum.ai/docs/test-cases): one session per arm, many scored rows. Plant defects with a patch that `setup` applies, anchor each defect's row on the identifier a finding must cite, and add a distractor, correct code that looks wrong, checked with `transcript_omits`. Without it, reporting everything passes every row. ## Spend little, then enough Keep runs cheap while you write, then raise the repetitions for the result you rely on: ```sh npx -y terum-skills@latest eval my-skill --triggers-only npx -y terum-skills@latest eval my-skill --case blocked-release npx -y terum-skills@latest eval my-skill --k 3 ``` `--triggers-only` runs no agent sessions. `--case` runs one of your case files, the cheap way to check that a new one loads. `--no-gen` stops a missing kind from being generated. Run `--k 3` for a receipt you intend to gate on. At the default of 1 each case is a single draw, and a verdict on a band edge flips easily: six rows at two wins and four ties is exactly +1/3 and `PASS`, and one win coming back a tie makes it `NEUTRAL`. Then [read the run](https://www.terum.ai/docs/results), starting with any check that failed, or passed, in both arms. ## Checklist - Every check is one an agent without the skill could pass by doing the job, and each "should" has a "should not" beside it. - Every task is a plain request an agent can finish alone, silent about the answer. - Identifier anchors, one behavior per case, and no check that neither arm can pass. - A rubric only where quality decides, carrying its own facts. - Trigger prompts phrased several ways, and near misses from competing skills. - An `adversarial` case, every host tool in `requires`, and a `--k 3` run before you gate.