# Running evals One command evaluates a skill folder on your machine against a baseline, and against your team's last evaluated version when there is one. This page covers what a run does, the options worth knowing, and queueing runs overnight. ```sh npx -y terum-skills@latest eval ``` The ref is a skill name or a path to its folder, and either must be in your Library: Global (`~/.claude/skills`) or a project you registered with `project add`. A team is optional. It adds the incumbent arm, the license policy hygiene checks against, and a place to share the receipt. ## One run, in order ::: steps ### Resolve the skill The ref is matched against your Library roots. On a machine with more than one team configured, name one with `--team`. ### Refresh the team clone With a team, its clone is fetched and hard reset to `origin/main`, fixing the incumbent and team policy for the run. ### Hygiene The same deterministic checks `validate` runs. An error ends the run before any model call. See [Hygiene checks](https://www.terum.ai/docs/hygiene). ### Dependencies and the heavy question Files that `SKILL.md` references elsewhere in its git repository are staged with the skill, up to 20 MB in total. If the skill looks like it launches subagents, the run asks `Use the one-session heavy evaluation mode?`. The answer is recorded, but what actually decides one session per arm is whether the run has an `evals/suite.yaml`, authored or generated. ### Preflight `claude --version`, then one real one-turn task in a throwaway directory. A failure stops the run before generation or any case. ### Generate what is missing Missing cases (or a suite) and trigger prompts are written into the skill's `evals/`, unless you pass `--no-gen`. Authored files are never overwritten, and `evals/` is outside the content digest, so the skill's identity and score do not change. Trigger prompts then run as tool-free model calls. See [Generated evals](https://www.terum.ai/docs/generated-evals). ### Choose the arms `baseline` (nothing staged) and `candidate` (your folder) always run. `incumbent` joins when the team has a receipted version with different bytes: the one with the most recent receipt, not the highest version number. ### Seed and run each arm Each arm and repetition gets a fresh directory: the case's fixture, inline files and `setup` script, then the skill at `.claude/skills/` without its `evals/` and `fixtures/`. The baseline gets no skill. ::: ## What an arm is Each arm is one headless Claude Code session, `claude -p` with the case's task, running in its sandbox on your own login. Every arm gets the same allowed tools and the same appended note: this is an unattended evaluation, no human can answer questions, so act on your best judgment and finish. `Task` and `Workflow` are on the list so a skill that launches subagents is not blocked. Your `~/.claude/settings.json` is not loaded. After each session, the engine reads which skills the session resolved and refuses the whole eval if the skill under test is missing from an arm that staged it, or present in one that did not. Sandboxes are plain directories, not containers: the skill's code runs on your machine. ## Retries and dead arms Each case, arm and repetition gets at most two attempts. A session that exits non-zero runs once more in a fresh sandbox; one that hits its time cap (120 minutes unless the case sets another) is not retried. A case whose setup fails is dropped, and the run continues. An arm that fails twice or times out is dead. Its rows are an unscored hole, never a zero: they are left out of every count and mean, and show as the gap between expected and scored rows, which marks the run partial. See [Results and receipts](https://www.terum.ai/docs/results). ## Options that matter | Option | Effect | | --- | --- | | `--k ` | Repetitions per case. Default 1. Use `--k 3` for a receipt you intend to gate on. | | `--case ` | Run one authored case. Skips an authored suite. | | `--triggers-only` | The trigger eval only: no sandboxes, no arms. The cheapest useful signal. | | `--execution-only` | The execution cases only. Cannot be combined with `--triggers-only`. | | `--model ` | The model for every arm, trigger selection and generation. Default `sonnet`. | | `--judge-model ` | The model for the tie-breaking judge. Defaults to `--model`. | | `--no-gen` | Generate nothing. A folder with no eval assets then evaluates nothing. | | `--no-commit` | Keep the receipt on this machine. | ::: warning Every arm is a real Claude Code session billed to your Claude account, and no spend cap exists. A run of ordinary cases is cases × k × arms sessions: five cases at `--k 3` with an incumbent is 45. ::: ## Batches and the overnight queue ```sh npx -y terum-skills@latest eval deploy-check hybrid-review handoff ``` Every ref is resolved before any paid work, and one preflight covers the whole run. Skills run four at a time by default (`--parallel `). `--batch ` asks before each further batch, and `--pending` adds every shared skill whose current version has no receipt. Add `--window overnight` and the same command queues the skills instead, spending nothing. Each queued item records the folder's content hash, so bytes that changed after queueing are refused rather than billed, and bytes that already have a local receipt are skipped. Only the desktop app drains the queue by itself. With **Settings ▸ Evals ▸ Run queued evals overnight** on (the default), it fires at most once a night, between 01:00 and 05:00 local time, while the app is open, idle for 30 minutes, and running nothing else. If anything is queued for overnight, it drains the whole queue, four at a time. With the app closed, the queue waits for someone to run `eval --drain`. ## From the app On a skill's page, **Run eval** opens a dialog with the cost estimate, when there is one, and the equivalent terminal line; from there you run now or **Queue for overnight**. In the Library, **Select** then **Evaluate N skills…** runs several. The app runs one eval at a time, with a chip and a **Stop** button in the top bar. Closing the dialog does not stop a run; quitting the app does.