Generated evals

#When generation happens

Each kind of asset is decided on its own, and only a missing kind is generated:

• Execution assets, when the folder holds no case file and no evals/suite.yaml.
• Triggers, when the folder holds no evals/triggers.yaml.

An asset that exists is never overwritten, and no flag forces regeneration. --no-gen turns generation off, and --case <stem> never generates a case.

Generation uses the run’s model (--model, default sonnet) on your own subscription: one tool-free call per kind, and at most three if answers fail validation.

#Cases or a suite

The model chooses the shape. The prompt asks for a suite when the skill has to recover a hidden ground truth, such as planted defects for a skill that reviews or fixes code. It asks for cases when the skill’s behavior depends on how it is asked, one case per behavior, and for cases with a judge rubric when neither fits.

To fix the shape yourself, set it in the SKILL.md frontmatter. suite and cases are the only values:

metadata:
  eval:
    shape: suite

#Generated cases

The generator picks between 3 and 7 cases from the skill’s complexity; the count is not a flag. Every case carries a bucket, and at least one must be adversarial. Scoring never reads buckets: they label the set for whoever reviews it.

Generated checks may use every check kind except command_succeeds. That one would make the generator write a verification program, and a wrong verifier silently corrupts the score in both arms. A generated case has no fixture; it seeds its sandbox from inline files and setup alone.

Before anything is written, each case is seeded once in a throwaway sandbox and its setup actually runs. That catches prose in setup, which is valid shell: “Assume codex is logged in” runs a program named Assume, exits 127, and would have dropped the case at run time.

#Generated suites

A generated suite is a small repository with an answer key: a clean base, a diff that plants between 2 and 6 defects and exactly one distractor, and one shell probe per defect. Each defect row checks transcript_mentions on an anchor, such as an identifier a correct finding has to cite. The distractor row checks transcript_omits.

The model never writes shell that runs against an arm. The engine composes the setup: it commits the clean base, runs every probe (each must pass), applies the diff, runs every probe again (each must now fail), and deletes the probes and the diff. An arm opens on a repository whose only uncommitted change is the planted diff. A probe that disagrees with the diff aborts the suite, so nothing is scored against a defect that is not there. The whole setup is dry-run once before the suite is written.

#Triggers

A generated trigger file has exactly five should_trigger and five should_not_trigger prompts. The model sees the catalog of sibling skills in the same Library root, so its near misses can come from the skills yours competes with.

#Where the files land

Generated files go straight into your skill folder, at evals/cases/<name>.yaml, evals/suite.yaml and evals/triggers.yaml, each opening with a comment that marks it as generated. They pass the HYG2 and HYG3 checks first; a finding writes nothing. The run also keeps its own copy.

evals/ is excluded from the content digest, so generated assets do not mint a version and do not blank a score: receipts of the same skill bytes still match. The line eval prints before writing says the next publish mints a new version and the local score blanks. Neither happens.

To regenerate, delete the asset and run eval again. The shape is chosen afresh, so deleted cases can come back as a suite.

#Review before you trust them

The generator reads your whole SKILL.md, so it can write cases that reward restating the rules rather than doing the job. Nothing in the engine prevents that; a person reading the cases does. Check two things first:

• Does each case test something the skill has to get right, or only that its vocabulary appears in the transcript?
• Is each should_not_trigger prompt a genuine near miss?

Both failures are invisible in the verdict line. For a suite, also check that each defect is a bug a maintainer would fix, that each anchor is something a correct finding must cite rather than a phrase the skill could hit by luck, and that the distractor is genuinely correct code. A row that marks correct code as a defect turns every arm that reports it into a loss.

Once you are happy with them, publish the skill and the reviewed assets go to your team with it. Writing good evals covers editing them by hand.