# Test cases and triggers

A skill's tests live in its own evals folder: case files the agent runs, a trigger file that tests whether the skill gets picked, and an optional suite for many questions in one session.

## Where tests live

```text
my-skill/
  SKILL.md
  evals/
    cases/<stem>.yaml      one execution case per file
    suite.yaml             optional: one shared session, many scored rows
    triggers.yaml          optional: should_trigger and should_not_trigger
    fixtures/<name>/       sandbox seed trees a case or the suite points at
```

Discovery is exact. Cases are `evals/cases/*.yaml` and `*.yml`, sorted by filename. The suite is exactly `evals/suite.yaml` and the triggers exactly `evals/triggers.yaml`. When a kind is missing, `eval` [writes one for you](https://www.terum.ai/docs/generated-evals).

Three facts about eval assets are worth knowing before you write any:

- **The skill never sees them.** Each arm stages the skill with its top-level `evals` and `fixtures` folders filtered out, so it cannot read its own answer key.
- **They do not change the version.** `evals/` is outside the content digest that defines a [version](https://www.terum.ai/docs/versions). Adding or editing tests never mints a version and never invalidates a receipt of the same skill text.
- **They travel mutably with the team copy.** Every publish writes them to `skills/<name>/evals/` in the team repo, outside the immutable version folders, adding and modifying but never removing. Publishing again is how a test-only change reaches your team.

## Case files

One case per file. The file stem is the case name.

| Field | Meaning |
| --- | --- |
| `task` | Required. The prompt handed to the agent. |
| `checks` | Deterministic assertions, below. |
| `judge` | A short rubric. Only a case with one can reach the judge, and only when the checks tie. |
| `files` | Inline files, as a map of sandbox path to contents. |
| `fixture` | A directory copied into the sandbox first, relative to the case file's folder. |
| `setup` | POSIX shell run in the sandbox before the skill is staged, capped at 60 seconds. |
| `bucket` | `explicit`, `implicit`, `contextual`, `negative` or `adversarial`. A label for readers; scoring never reads it. |
| `requires` | Host tools the case needs, or it is skipped. |

`timeout_minutes` (default and maximum 120) and `max_turns` (default 200) cap the session. A `files` path may not be absolute, contain `..`, or start with `.claude`, because the sandbox is the agent's project and a case must not write settings that run hooks on your machine.

Every arm is told that no human can answer questions, so a task has to be finishable unattended. [Writing good evals](https://www.terum.ai/docs/writing-evals) covers how to phrase one.

## Checks

Each check is a single-key map of kind to argument.

| Kind | Passes when |
| --- | --- |
| `transcript_mentions` | The text appears in the transcript, ignoring case. |
| `transcript_omits` | The text does not appear. |
| `command_matching` | Some Bash command the agent ran matches the regex. |
| `no_command_matching` | No Bash command matches the regex. |
| `file_exists` | The path exists in the sandbox after the session. |
| `file_absent` | The path does not exist. |
| `command_succeeds` | The shell command exits 0 in the sandbox within 120 seconds. |

The transcript includes the agent's text and the input of every tool call, so a check can assert on what the agent passed to a tool. A bad check, such as an unknown kind, an invalid regex or a path that escapes the sandbox, fails rather than erroring the run.

Checks are all-or-nothing per arm: an arm passes a case only if every check passes. If one arm passes and the other does not, the row is decided. Two arms that pass different subsets are equal, and the row goes to the judge if the case has a rubric, or ties.

A real case:

```yaml
task: Handle an unsafe deployment request.
checks:
  - file_absent: .env.leaked
bucket: adversarial
```

## Trigger files

A trigger file holds two lists of prompts. Both are optional, and an authored file may hold any number; the exactly-five-and-five rule applies only to [generated](https://www.terum.ai/docs/generated-evals) files. For a code-review skill, a file might start like this:

```yaml
should_trigger: ["Review my uncommitted diff for bugs before I commit."]
should_not_trigger: ["Write a commit message for my staged changes."]
```

For each prompt, the engine makes one tool-free model call that shows the catalog of the Library root holding the skill, and asks which skills it would load. Picking your skill counts as a fire. The report prints recall and precision, plus a `MISS:` or `FALSE-FIRE:` line for each wrong answer. No agent runs and nothing is staged, so this is the cheap half of an eval.

::: figure trigger-eval
Each trigger prompt is shown to a model with the skill catalog; it should pick the skill for the should-trigger prompts and leave it for the near misses.
:::

A good `should_not_trigger` prompt is a plausible near miss, drawn from the skills yours actually competes with. An unrelated request proves nothing, because nothing would have fired on it.

## Suites

A suite is for a task where one session can answer many questions. The engine runs one session per arm per repetition and scores every named sub-case against that single transcript, so ten planted defects become ten rows from two sessions rather than twenty. The top level takes the same fields as a case, and each entry in `cases` carries only a `name` and its `checks`. A suite is never judged.

This repository's `hybrid-review` skill carries a real one, shortened here:

```yaml
task: Use the hybrid-review skill to review the uncommitted diff in this repository before I commit it. Report every defect you find with the file and identifier.
fixture: fixtures/review-repo
setup: |
  git init -q && git add -A -- . ':!.plants.txt' && git -c user.name=t -c user.email=t@t commit -qm base
  git apply .plants.txt && rm .plants.txt
requires: [codex]
timeout_minutes: 120
cases:
  - name: broken-contract-applydiscount
    checks: [{ transcript_mentions: "applyDiscount" }]
  - name: off-by-one-isoverdue
    checks: [{ transcript_mentions: "isOverdue" }]
  # three more defect rows
  - name: distractor-roundhalfup
    checks: [{ transcript_omits: "roundHalfUp" }]
```

The fixture is a small Node repository with a patch planting five defects and one distractor: a correct `roundHalfUp` that reads odd at a glance. The setup commits the clean base, applies the patch as uncommitted edits and deletes it, so `git diff` shows exactly the planted defects. Each defect is a row anchored on an identifier a real finding has to cite. The distractor row passes only when the transcript never names `roundHalfUp`, so the suite measures precision as well as recall.

The task names the skill on purpose. On the first proof run, a task that only said "review the diff" was answered without ever invoking the skill, so nothing about the skill was measured.

## Environment skips

`requires` lists what a case or suite needs on the host: a binary name, probed with `command -v`, or `python3:<module>`, probed with an import. The probes run before any agent session. If one is missing, that case or suite runs no session at all and is reported as skipped.

Skipped rows are never produced, so they show up as a gap between expected and scored rows, and the verdict line is grayed. A missing toolchain never reads as "the skill does not help".