Evaluations

Hygiene checks

Eight deterministic gates read a skill folder before any model is called. They cost nothing, run offline, and stop an eval or a publish on any error.

What hygiene is

Hygiene is the free tier of evaluation. It is deterministic: it reads a skill folder’s bytes in memory, calls no model, and touches no network.

Findings come in two tiers. An error fails closed: whatever was about to happen, an eval or a publish, stops. A warning prints and gates nothing.

A skill folder passes eight deterministic gates before anything is paid for; an error stops it, and a warning only prints.

The eight gates

CodeWhat it checksTier
HYG1SKILL.md frontmatter against a strict schema, name against the folder name, and the allowed-tools grantError
HYG2Bidi controls, zero-width characters, and words that mix Unicode scriptsError
HYG3Credential-shaped values, and email addresses that are not the author’sError
HYG4Executable mode bits, a leading #!, and file extensions outside the allowlistError
HYG5Agreement between the declared license, the team’s license policy, and any bundled LICENSE* fileError
HYG6An empty description (error), and a SKILL.md over 20,000 characters, roughly 5,000 tokens (warning)Both
HYG7metadata.terum-category against the team’s category listWarning
HYG8Paths in the SKILL.md body that point outside the skill folder, into the rest of the repositoryWarning

For HYG3, addresses at reserved domains such as example.com are exempt, as is the one in angle brackets in your metadata.author field. Use a reserved domain in fixture data.

Two gates that surprise people

Mixed scripts (HYG2). The rule applies per whitespace-delimited word, not per file. Multilingual prose switches alphabets between words, so it passes. A homoglyph attack, such as a Cyrillic letter hidden inside a Latin word, switches inside one word, so it fails. Digits, punctuation and symbols are always allowed. The honest casualty is a hyphenated cross-script word such as GitHub-репозиторий, which you rewrite.

The extension allowlist (HYG4). Only these extensions may appear, a file with no extension fails, and the list is never waived:

.md .txt .json .yaml .yml .csv .toml .xml .html .css .js .ts .py .sh .sql .svg .png .jpg .jpeg .gif .webp .pdf

Separately, an executable mode bit or a #! first line fails, even on a .sh file: the executable form is the signal, not the language. publish waives those two findings and carries the file and its mode to the team. validate, eval and skill fix never waive them, so a folder that publishes can still fail validate.

Validate a folder

npx -y terum-skills@latest validate <path|name>

validate runs the same gate, offline. Warnings print first, one line each prefixed with warning. Each error prints as HYG<n> <path>[:<line>]: <message>. A clean folder prints my-skill: hygiene passed. The exit code is 0 with no errors and 1 with any; warnings never change it.

Without --cwd, a directory at the path you give wins. Only when nothing is there is the argument treated as a skill name in your team’s clone, resolved to its newest version.

Two things catch people out:

  • Without --cwd, validate needs a configured team, because it reads the team’s license policy from your clone. --cwd <checkout> validates against a team checkout directly; the flag exists for CI checkouts that have no configuration.
  • A folder that has never been published fails HYG1. validate requires the four managed fields that publish writes (license, metadata.id, metadata.author and metadata.terum-category). eval treats those four as optional, so an unpublished folder can be evaluated but not validated.

Repair with skill fix

npx -y terum-skills@latest skill fix <path>

skill fix repairs only faults with exactly one right answer. In order, it:

  1. Quotes a frontmatter value that YAML refuses because it contains : , leaving the text unchanged.
  2. Sets name: to the folder name.
  3. Sets license: to the team’s policy, when a team is configured and the declared license differs.
  4. Removes every HYG2 invisible character from every text file.
  5. Clears the executable bit on any file that is not a script. A file starting with #! keeps its bit.

Each change prints one sentence. Then it runs hygiene again and lists what is left under Still needs you. Anything that takes judgment is reported, never repaired: a description, a category, an extra frontmatter field, a credential-shaped value, a mixed-script word, or a shebang script. Nothing is moved and nothing is published.

When hygiene runs

  • validate: the folder on disk, with the managed fields required.
  • eval: the local folder as it is, before any model call, with the managed fields optional. An error refuses the run with Hygiene failed for <name>: and the findings.
  • publish: the folder after the managed fields are injected, with the mode-bit and shebang findings waived. HYG7 fires here and only here, because only publish has the team’s category list.
  • skill fix: the folder after its repairs.

When eval generates eval assets, the new files also pass through the HYG2 and HYG3 checks before they are written. A finding aborts the write, so a clean run never leaves your folder dirty.

Full reference on GitHub