# Generated evals

When a skill folder has no tests, eval writes the missing ones into it with the run's own model. They are yours to review before you trust the verdict, and to publish with the skill.

## When generation happens

Each kind of asset is decided on its own, and only a missing kind is generated:

- **Execution assets**, when the folder holds no case file and no `evals/suite.yaml`.
- **Triggers**, when the folder holds no `evals/triggers.yaml`.

An asset that exists is never overwritten, and no flag forces regeneration. `--no-gen` turns generation off, and `--case <stem>` never generates a case.

Generation uses the run's model (`--model`, default `sonnet`) on your own subscription: one tool-free call per kind, and at most three if answers fail validation.

## Cases or a suite

The model chooses the shape. The prompt asks for a suite when the skill has to recover a hidden ground truth, such as planted defects for a skill that reviews or fixes code. It asks for cases when the skill's behavior depends on how it is asked, one case per behavior, and for cases with a `judge` rubric when neither fits.

To fix the shape yourself, set it in the `SKILL.md` frontmatter. `suite` and `cases` are the only values:

```yaml
metadata:
  eval:
    shape: suite
```

## Generated cases

The generator picks between 3 and 7 cases from the skill's complexity; the count is not a flag. Every case carries a bucket, and at least one must be `adversarial`. Scoring never reads buckets: they label the set for whoever reviews it.

| Bucket | The case asks |
| --- | --- |
| `explicit` | The user names the job the skill is for. |
| `implicit` | The user describes the job without naming it. |
| `contextual` | The right behavior depends on the state of the sandbox, not the wording. |
| `negative` | The skill should hold back, refuse, or do less than it could. |
| `adversarial` | The request pushes at the skill's rules, and the correct answer is not the obliging one. |

Generated checks may use every [check kind](https://www.terum.ai/docs/test-cases#checks) except `command_succeeds`. That one would make the generator write a verification program, and a wrong verifier silently corrupts the score in both arms. A generated case has no `fixture`; it seeds its sandbox from inline `files` and `setup` alone.

Before anything is written, each case is seeded once in a throwaway sandbox and its `setup` actually runs. That catches prose in `setup`, which is valid shell: "Assume codex is logged in" runs a program named `Assume`, exits 127, and would have dropped the case at run time.

## Generated suites

A generated suite is a small repository with an answer key: a clean base, a diff that plants between 2 and 6 defects and exactly one distractor, and one shell probe per defect. Each defect row checks `transcript_mentions` on an anchor, such as an identifier a correct finding has to cite. The distractor row checks `transcript_omits`.

::: figure suite-defects
A generated suite plants defects and one distractor in a small repository; each defect the agent names passes a row, and naming the distractor fails one.
:::

The model never writes shell that runs against an arm. The engine composes the setup: it commits the clean base, runs every probe (each must pass), applies the diff, runs every probe again (each must now fail), and deletes the probes and the diff. An arm opens on a repository whose only uncommitted change is the planted diff. A probe that disagrees with the diff aborts the suite, so nothing is scored against a defect that is not there. The whole setup is dry-run once before the suite is written.

## Triggers

A generated trigger file has exactly five `should_trigger` and five `should_not_trigger` prompts. The model sees the catalog of sibling skills in the same Library root, so its near misses can come from the skills yours competes with.

## Where the files land

Generated files go straight into your skill folder, at `evals/cases/<name>.yaml`, `evals/suite.yaml` and `evals/triggers.yaml`, each opening with a comment that marks it as generated. They pass the [HYG2 and HYG3 checks](https://www.terum.ai/docs/hygiene) first; a finding writes nothing. The run also keeps its own copy.

`evals/` is excluded from the content digest, so generated assets do not mint a version and do not blank a score: receipts of the same skill bytes still match. The line `eval` prints before writing says the next publish mints a new version and the local score blanks. Neither happens.

To regenerate, delete the asset and run `eval` again. The shape is chosen afresh, so deleted cases can come back as a suite.

## Review before you trust them

The generator reads your whole `SKILL.md`, so it can write cases that reward restating the rules rather than doing the job. Nothing in the engine prevents that; a person reading the cases does. Check two things first:

- Does each case test something the skill has to get right, or only that its vocabulary appears in the transcript?
- Is each `should_not_trigger` prompt a genuine near miss?

Both failures are invisible in the verdict line. For a suite, also check that each defect is a bug a maintainer would fix, that each anchor is something a correct finding must cite rather than a phrase the skill could hit by luck, and that the distractor is genuinely correct code. A row that marks correct code as a defect turns every arm that reports it into a loss.

Once you are happy with them, publish the skill and the reviewed assets go to your team with it. [Writing good evals](https://www.terum.ai/docs/writing-evals) covers editing them by hand.