Eval files¶

Each YAML eval file has a top-level cases: list. Unknown keys within a case or an assertion are rejected — a typo like assertion: would otherwise produce a case that passes vacuously. Extra keys alongside cases: at the top level of a YAML file are ignored, with one exception skill-lens reads: tool_libraries: — see Sharing tools across eval files. A skill can also keep its cases in an evals.json, which has no cases: list and is strict at every level — see Reading evals.json.

Field	Required	Meaning
`name`	yes	Case name, shown in reports. Unique among the skill's cases, ignoring letter case — see Names are unique
`task`	one of `task` / `sessions`	The prompt handed to the runner
`sessions`	one of `task` / `sessions`	Several prompts, each a fresh session in one shared workspace — see Several sessions in one workspace
`assertions`	no	Scoring rules; a case with none passes
`tags`	no	Labels for `--tag` filtering
`tools`	no	Mock tools the agent may call — see Mock tools
`trajectory`	no	Which tools must/must not have been called, in what order, and with what arguments — see What a tool was called with
`budget`	no	Ceilings on tokens, cost, and latency
`judge`	no	A rubric for an LLM judge — see Judging output quality
`workspace`	no	A temporary directory and the files it starts with
`mode`	no	`loaded` (default) or `offered` — see Did the agent reach for the skill?

Choosing a check¶

Four kinds of check, and each answers a different question.

Use	When you want to know	Costs
`assertions:`	Did the output — or a file the agent wrote — contain the right text, match a pattern, or validate against a JSON schema?	Nothing
`trajectory:`	Did the agent reach for the skill, and call the right tools — in the right order, with the right arguments, no more often than it should?	Nothing
`budget:`	Did it stay inside a token, cost or latency limit?	Nothing
`judge:`	Is the output good — complete, correctly reasoned, in the right tone?	One judge call per case that carries a rubric

Reach for the cheapest one that answers your question. An assertions: entry is exact and free; a judge: rubric is the only thing that can grade quality, and every case carrying one spends a judge call — so it has to be turned on, because the default judge = "fake" grades nothing and reports such a case as errored. A skill whose job is to call something is measured by trajectory:, not by what it said about calling it.

Some rules that catch people out:

A rubric entry phrased against a mock tool's returns: is an authoring error. The judge never sees a tool's return, so that entry is unverifiable — reword it, or name a workspace: file under judge.artifacts.
In a judge: rubric only, a check the judge passes without citing evidence is recorded as a failure. Assertion, trajectory and budget checks build their evidence from the same comparison that produced the verdict, so the rule does not apply to them — see Per-check results.
A trajectory: naming a tool the case does not declare is refused before any case runs, under fake, pydantic-ai and langchain. A product's own tools cannot be listed, so copilot and claude-code refuse no name, and cli refuses a trajectory: block outright.

Workspaces¶

A workspace: block gives one case a real, contained temporary directory: created fresh for that work item, seeded with the files it declares, and deleted once the case is scored — unless --keep-workspace says otherwise.

    workspace:
      files:
        sales.csv: |
          region,units
          north,120
          south,80

It is opt-in. A case with no workspace: block gets no temporary directory and no extra tools, so every suite written before this existed keeps running byte-identically. Declaring workspace: {} with no files: is still meaningful — it hands the agent an empty directory to generate something into, for a skill whose whole job is producing a file from nothing.

With the block present, the agent also gets three built-in tools:

Tool	Does
`list_files`	List every file in the working directory, one relative path per line
`read_file`	Read a text file. `path` is relative to the working directory
`write_file`	Create or replace a text file

A skill that ships any file beside SKILL.md besides its eval files also gets list_skill_files, read_skill_file and — only when the run enables it with allow_scripts or --allow-scripts — run_script. See Bundled files and scripts. All six names are reserved in any case with a workspace: block, whether or not the skill has a bundle, so a case's own tools: may not declare a name that collides with one of these six — that is an authoring error. A trajectory: block may name any of them like any other tool:

    trajectory:
      called: [run_script, write_file]

Naming a built-in in a case with no workspace: block is an authoring error too: the bundle tools exist only where the workspace does.

Like the offered-skill tool below, a call to one of these six lands in the trajectory like any other tool call, so it counts toward max_calls and any budget: ceilings too.

See The workspace for containment and the size caps, and Assertion kinds below for scoring a produced file rather than the chat output.

Several sessions in one workspace¶

Some skills claim that a new session can pick up from files an earlier session wrote: a plan, a progress log, a memory file. One task: cannot test that. Seeding the files by hand does not test it either: a case that hands the agent a finished plan tests reading a file the author wrote, and it passes with or without the skill.

sessions: replaces task: with a list of tasks. Each one is a separate session: a fresh runner call, with no conversation history, against the same workspace. The case is scored once, after the last session.

cases:
  - name: resumes-from-its-own-plan
    sessions:
      - >-
        Plan the migration of the billing service to the new queue in three phases.
        Do phase 1 only, then stop.
      - Continue the migration from where it stopped.
    workspace: {}
    assertions:
      # Session 1 was told to stop after phase 1, so a progress file that marks
      # phase 2 done is progress only a resuming session could make.
      - kind: regex
        value: '(?im)phase 2.*(done|complete)'
        file_glob: .planning/*/progress.md
    judge:
      rubric:
        - progress.md records phase 2 as done, after phase 1, in the same plan
      artifacts:
        - glob: .planning/*/progress.md

Check what only a later session could do. Every check runs after the last session, so it cannot tell which session wrote a file. file-produced on a file the first session creates passes even when the second session did nothing at all. Check for progress the first session was told not to make, as the example does. Because a file_glob must match exactly one file, the same check also fails when the second session starts a new plan in a new directory instead of continuing the old one. A first session that ignored "then stop" would still pass the regex; the rubric's "after phase 1, in the same plan" is there for the judge to read the file as a whole.

What each part of the case reads:

Part	Reads
`assertions:` on the output	The last session's output
`assertions:` with `file:` / `file_glob:`	The workspace after the last session
`trajectory:`	Every session's tool calls, in order — so `forbidden:` and `max_calls` cover every session, and `order:` may span sessions
`budget:`	The sum over every session: tokens, cost and latency
`judge:`	Every session's task, the last session's output, and the artifacts after the last session
`mode: offered`	Every session is offered the skill, and `trajectory.skill_triggered` must hold in every session: `true` fails when any session did not load the skill, `false` fails when any session did. In the report, `skill_triggered` is true when any session loaded it, and `skill_triggered_by_session` lists each session's own answer

The rules:

task: and sessions: are exclusive, and a case needs one of them.
sessions: needs at least two entries — one is task: spelled longer — and no entry may be blank.
sessions: needs a workspace: block. The workspace is the only thing that carries over from one session to the next; without one the sessions would be unrelated tasks.

Each of these is an authoring error: the run stops with exit 2 before any case runs.

Nothing but the workspace carries over. Each session builds its tools again, so a mock tool's returns: list starts from its first entry in every session, and a product runner delivers the skill again before each session. The workspace is seeded once, before the first session, and deleted once, after scoring.

If a session errors — a provider failure, a timeout, a product that crashed — the case is errored, and the sessions after it do not run. The report says which session failed (session 2 of 3: ...), and the spend of every session that ran is kept. A session that a keyed runner stopped at its token budget is the last to run too, though the case is failed on its budget rather than errored; each session's stop is what the earlier sessions left of max_tokens. --repeat repeats the whole sequence, each time in a fresh workspace, and the baseline arm runs the same sessions without the skill, in a workspace of its own.

Mock tools¶

A tools: list declares the tools the agent may call. Nothing executes: calling one records the call and returns returns verbatim, so the trajectory is the model's own choice and the run has no side effects. See Declaring tools and scoring the trajectory for how the calls are scored. The same block serves a product runner: copilot and claude-code get the tools through an MCP server skill-lens starts for the case, under the same names, with the same canned returns.

A tool declares its arguments one of two ways:

    tools:
      - name: lookup_order              # ^[A-Za-z0-9_-]{1,64}$ — what providers accept
        description: Look up an order by its id
        parameters:                     # the shorthand: name -> primitive type
          order_id: string              # string | integer | number | boolean
        returns: '{"id": "1234", "status": "delivered"}'
      - name: get-pull-request
        description: Get details of a specific pull request
        input_schema:                   # a full JSON Schema, passed to the agent verbatim
          type: object
          properties:
            owner: {type: string}
            pull_number: {type: integer}
            labels: {type: array, items: {type: string}}
          required: [owner, pull_number]
        returns: '{"number": 1}'

parameters: is for a tool you describe by hand. Every key is required and the schema is closed (additionalProperties: false), because you wrote every key. input_schema: is for a tool that must match a real server's declared schema — optional arguments, enums, arrays, nested objects — and reaches the agent exactly as written; nothing is added. It must be a valid JSON Schema whose top-level type is object, which is what every provider requires of a tool. A tool that declares both, or an input_schema that is not a valid object schema, is an authoring error (exit 2) caught before any case runs.

skill-lens mcp-import writes an input_schema: block from an MCP server's own tools/list listing, so a mock for a real server's tool is copied rather than transcribed.

Tool names follow the rule both OpenAI and Anthropic enforce, ^[A-Za-z0-9_-]{1,64}$, so a hyphenated MCP tool name such as get-pull-request is kept as the server spells it. A name outside the rule is an authoring error; skill-lens never rewrites one.

Answering differently per call¶

One returns: string answers every call the same way. A skill whose instructions loop over a tool — fetch work item A, follow its parent link, fetch B, stop when there is no parent — needs the mock to answer differently, and returns: takes two more shapes for that. The shape says which rule applies.

A list of strings is a sequence, consumed in the order the calls arrive: the first call gets the first entry, the second call the second, and every call after the list is used up gets the last entry again.

    tools:
      - name: get_work_item
        description: Fetch a work item by id, with its parent link
        parameters:
          id: string
        returns:
          - '{"id": "A", "parent": "B"}'
          - '{"id": "B", "parent": null}'      # the third call and every later one get this too
    trajectory:
      called: [get_work_item]
      max_calls: 2                            # the check for a loop that should have stopped

The last entry repeating is the steady state a skill that keeps calling should see — the item with no parent, the job that is done. It is not a check on how many calls were made: trajectory.max_calls is. A sequence ignores the arguments, so it is the right shape when the position of the call decides the answer (polling until done) and the wrong one when the argument does: a model that issues several calls in one turn gets the entries in whatever order the framework runs them.

A list of when:/value: mappings is a lookup, answered by the first entry whose when: keys all equal the call's arguments — a subset is enough; the call may carry more arguments than when: names. An entry with no when: matches every call, which makes it the fallback.

    tools:
      - name: get_work_item
        description: Fetch a work item by id, with its parent link
        parameters:
          id: string
        returns:
          - when: {id: "A"}
            value: '{"id": "A", "parent": "B"}'
          - when: {id: "B"}
            value: '{"id": "B", "parent": null}'
          - value: '{"error": "not found"}'   # no when: -- every other call lands here

when: matches by the same rule as call_args contains:: a subset at every level (a nested mapping may name only the keys that matter), a list element by element at equal length, and values as YAML and JSON parse them — when: {id: 1} matches a call with the integer 1, when: {id: "1"} a call with the string "1", and a true matches only a boolean, never a 1. Without a fallback, a call that matches nothing gets the fixed reply no response is scripted for get_work_item with arguments {"id": "C"} (the arguments as sorted JSON): the skill asked for something the case did not anticipate, which the transcript then shows, and a mock tool never raises.

Three lookup mistakes are authoring errors (exit 2), caught before any case runs, because each is a check that could never fire: a when: key the tool can never carry (one outside parameters:, or outside a closed input_schema — additionalProperties: false with its properties listed; an open schema may key on any name), an empty when: {} (drop the key to declare a fallback), and an entry an earlier one already answers — a fallback above it, the same when:, or a when: it only narrows — since the first match wins. So are an empty list (returns: '' is how an empty reply is spelled) and a list that mixes strings with mappings.

A ref: may set returns: in any of the three shapes; a lookup's when: keys are checked against the parameters the library declared.

Several skills often front one API — one MCP server behind nine skills — and every eval file that mocks get_pull_request would otherwise carry the same block. A tool library declares a tool once. It is a YAML file with one top-level tools: list, the same block a case takes, and exactly what skill-lens mcp-import prints — so skill-lens mcp-import tools.json > shared-tools/server.yaml writes one:

# shared-tools/order-api.yaml
tools:
  - name: lookup_order
    description: Look up an order by its id
    parameters:
      order_id: string
    returns: '{"id": "0000", "status": "delivered", "days_since_delivery": 0}'

An eval file imports libraries with tool_libraries: beside cases:, and a case names a tool with ref::

tool_libraries:
  - ../../shared-tools/order-api.yaml    # relative to this file; a directory imports every .yaml/.yml in it

cases:
  - name: refuses a refund outside the return window
    task: I want a refund for order 1234
    tools:
      - ref: lookup_order                # the library's name, description and schema
        returns: '{"id": "1234", "status": "delivered", "days_since_delivery": 45}'

The library owns the tool's contract — name, description, parameters: or input_schema: — and the case owns the scenario: a ref: may set returns: (a string, a sequence or a lookup, as above) and nothing else. Any other key beside ref: is an authoring error naming it; a case that needs a different contract declares the tool inline, and a ref: may sit in the same list as inline tools. After loading, the case is exactly what it would have been with the library's block pasted in: trajectory: names, the six reserved built-in names and the mode: offered collision rule all read the resolved tool, and every runner sees a plain mock.

Library paths are relative to the eval file — never to the working directory, never absolute — so the same file loads identically under discovery, under --evals, under skill-lens list and from any checkout. A library is checked as an eval file is: an unfilled TODO(skill-lens), a tool name outside the rule, an unknown key, or an invalid schema is an authoring error (exit 2) naming the library file and the tool's position. So is an unresolvable ref: — no tool_libraries: key, a name no import declares (the message lists the names they do), a missing file — a name two imported files both declare (never resolved by position), or one file imported twice.

Judging output quality¶

Some things an assertion cannot check: "explains it plainly" is not a substring. A judge: block hands those to an LLM judge.

    judge:
      expected: A short, plain-language refusal that names the order id.
      rubric:
        - The reply names order 1234
        - The reply explains that the return window has closed

Field	Required	Meaning
`expected`	no	Free text describing what a good answer looks like
`rubric`	yes	One statement per line, each checked independently
`artifacts`	no	Workspace files the judge may read, graded alongside the output — see Workspaces

The judge returns one verdict per rubric entry, with the evidence for it. skill-lens derives the verdict and the score from those per-check results; the judge is never asked for a blended number, because an unsupported PASS hidden inside one is invisible.

Two rules follow from that, and both are mechanical rather than a prompt asking nicely:

A check that passes without citing evidence is recorded as a failure. An unsupported PASS is an LLM judge's characteristic failure mode.
A verdict whose checks don't line up with the rubric is errored, not failed — malformed structured output is an infra problem, not evidence about the skill.

An empty rubric, or a blank entry within one, is an authoring error: a check that verifies nothing would score as a pass nobody verified.

The judge sees the task, expected, the response, and the files artifacts names — nothing else. It is never shown what a case's mock tools: returned, or which tools were called: it grades text, and never invokes the skill under test. So a rubric line must never be a comparison against the mock data, however much it reads like a sensible hallucination check:

    tools:
      - name: pull_request_threads
        description: The review threads on a pull request
        returns: '{"threads": [{"author": "Alex Chen", "comment": "Rename this."}]}'
    judge:
      rubric:
        # Refused: the judge cannot see the mocked data
        - The summary does not invent any reviewer or comment not present in the mocked data

A check like that is unverifiable as written. A careful judge fails it as ambiguous — a red case that looks like a regression — and a lenient one passes it without looking, which is worse: a suite can stay green for months on a check no judge ever verified. skill-lens refuses the line at load time (exit 2) when it names the harness's own vocabulary for that data: mock/mocked followed by data, response, result, return, value, output or tool; tool/mock (with or without a possessive 's or ') followed by returned, returns, return value, response, result, output, data or value; or returned by the tool/mock. A bare mock ("proposes a mock for the HTTP client") or tool ("names the tool it would use") is not refused, so a rubric about a testing skill or about the response itself is untouched; if a line trips the rule for a reason of your own, reword it. The judge is told the same thing in its own instructions — that it was not shown what any tool returned, which tools were called, or any mock data, and that a check decidable only against those is a fail — so a line the vocabulary rule does not catch still fails honestly rather than passing unread.

Two fixes, depending on what the check is really about. If it is about the response, phrase it against the response: "The summary names Alex Chen as the reviewer." If it is really a comparison against the data, put the data where the judge can read it — a workspace: file named under artifacts — and phrase the check against that file:

    workspace:
      files:
        threads.json: '{"threads": [{"author": "Alex Chen", "comment": "Rename this."}]}'
    tools:
      - name: pull_request_threads
        description: The review threads on a pull request
        returns: '{"threads": [{"author": "Alex Chen", "comment": "Rename this."}]}'
    judge:
      rubric:
        - The summary names no reviewer absent from threads.json
      artifacts: [threads.json]

artifacts names workspace files the judge may read, so a rubric can grade the document a skill produced rather than the chat message about it:

    judge:
      expected: A short Markdown report with one line per region and a total.
      rubric:
        - The report states a total of 200 units.
      artifacts: [report.md]

Each artifact reaches the judge fenced against its own content — labelled with the name you gave it, but the content itself is read as data, never as instructions, even one that reads like it is trying to talk to the judge. artifacts on a case with no workspace: block, or naming a file no case could ever produce, is an authoring error.

An entry may be a pattern instead of a path, under the same exactly-one rule as file_glob:

    judge:
      artifacts:
        - glob: .planning/*/progress.md
      rubric:
        - progress.md has an entry for this session

The judge sees the artifact under its pattern. The path the agent chose never enters the prompt. No match is shown as (not produced); several as (matched N files; exactly one is required). The same text as a path and as a glob in one list is an authoring error.

Judging costs money, so it is opted into explicitly with judge = "pydantic-ai" (or "langchain", or an installed product — "copilot", "claude-code", "cli" — which needs no API key, but spends the product's own quota) in skill-lens.toml. The default judge = "fake" does not grade at all — and rather than passing a rubric it never checked, it reports the case as errored. Judge spend is reported as "judge overhead", separately from what the runs themselves cost, and never counts against a case's budget:.

Did the agent reach for the skill?¶

mode: offered stops force-loading the skill. Instead it is registered as a tool, named after the skill and described by its frontmatter description. If the agent calls it, it receives the skill's instructions and carries on; if it doesn't, it never sees them — so the triggering decision is a real choice, and an observable one.

  - name: reaches for the skill on a refund question
    mode: offered
    task: I want a refund for order 1234
    trajectory:
      skill_triggered: true

  - name: leaves an unrelated question alone
    mode: offered
    task: What's the capital of Egypt?
    trajectory:
      skill_triggered: false

Always ship the negative control. A suite of positives alone scores a skill that fires on everything at 100%.

Three things to know:

The offered tool call lands in the trajectory like any other, so it counts toward max_calls.
Check it with skill_triggered, not by naming it in called: — under fake, pydantic-ai and langchain that list only accepts tools the case itself declares.
The tool name is the skill's name normalised to what providers accept — ASCII letters, digits and _, at most 64 characters (order-support becomes order_support, café becomes caf_). A case tool that collides with it is an authoring error. This normalisation applies only to the offered-skill tool; a case's own tools keep their names.

Setting skill_triggered on a mode: loaded case is an authoring error too: a loaded skill is always in force, so the check could never be false. Running an offered case on a runner that does not support the mode is errored, never a quiet pass.

Under a product runner there is no offered tool: the skill sits in the product's own skill directory, the bare task is sent, and skill_triggered comes from the product's own load signal (Copilot's skill tool call — its skill.invoked event is only what a slash invocation emits — Claude Code's Skill tool call). A product with no such signal — cli — makes mode: offered an authoring error under that runner, never a silent false that would pass every negative control.

Which runners serve which case features¶

Case feature	`fake`	`pydantic-ai` / `langchain`	`copilot` / `claude-code`	`cli`
`assertions:`	yes	yes	yes	yes
`tools:` (mock tools)	yes	yes	yes — through the MCP bridge	authoring error
`trajectory:`	yes — names must be the case's own tools	yes — names must be the case's own tools	yes — the case's tools by their declared names, or the product's own tool names, unchecked; `max_calls` counts the product's own calls too	authoring error
`mode: offered`	yes	yes	yes	authoring error
`budget:`	yes	yes — `max_tokens` also stops the run while it spends	see Product runners	latency only
`workspace:`	yes	yes	yes — the product's working directory	yes
`sessions:`	yes	yes	yes — the skill is delivered again before each session	yes
`judge:`	yes	yes	yes	yes

An authoring error here is found in preflight and exits 2 before any case runs. Under cli, budget: max_tokens and max_cost_usd are declared limits the runner cannot measure, so each is a failing not evaluated check rather than an error; only max_latency_ms is evaluated.

Unfilled scaffolds¶

skill-lens init writes placeholder fields holding the literal TODO(skill-lens). Loading a case that still contains one is an authoring error: the run aborts with exit 2 naming the file, the case, and the field.

That is deliberate, and it is the loader's rule rather than the scaffolder's — a hand-written stub is refused the same way. An unfinished eval that ran would either pass while checking nothing or fail while saying nothing about the skill, and both are worse than a run that stops and tells you which field to fill in.

Comments are discarded before the check, so a file may discuss the token freely. Mapping keys are checked as well as values — workspace: files: is keyed by filename, and an unfilled filename would otherwise seed a file literally named after the placeholder.

Assertion kinds¶

`kind`	Passes when
`contains`	`value` appears in the output
`not_contains`	`value` does not appear in the output
`regex`	`value` matches anywhere in the output (`re.search`)
`equals`	the stripped output equals `value` exactly
`file-produced`	`file` exists in the workspace
`json-schema`	the output (or `file`) parses as JSON and validates against `json_schema`

Every assertion in a case must hold for the case to pass. An unsupported kind or a malformed regex aborts the run as an authoring error rather than being reported as a skill failure.

file is a modifier, not a kind of its own. Set it on contains, not_contains, regex or equals and that assertion reads the named workspace file instead of the run's output text:

    assertions:
      - kind: contains
        value: "north"
        file: report.md

file: (on any kind, file-produced and json-schema included) in a case with no workspace: block is an authoring error — there would be no file to look at, so the assertion could never hold.

A file: naming a file that was never produced fails the assertion, not_contains included: an unreadable file fails regardless of kind, rather than being read as an implicit "the value isn't there" — an author checking not_contains against a file the skill never wrote should not expect that to pass.

The same split applies to what the run put at the path. A file: that could never name a workspace file — empty, absolute, containing .. — is an authoring error and aborts the run. A well-formed file: whose target the workspace refuses to read — a symbolic link a bundled script planted that points outside the workspace, a FIFO, a symlink loop, or a file larger than max_file_bytes — fails the assertion, with the refusal as the check's evidence: that is the skill's doing, not the author's. See The workspace for the read rules.

Matching a file by pattern¶

Some skills write to a path the model chooses: planning-with-files puts its plan in .planning/<date>-<slug>/. file_glob is the pattern form of file, accepted on every kind that accepts file:

    assertions:
      - kind: file-produced
        file_glob: .planning/*/task_plan.md
      - kind: regex
        value: "(?m)^## Goal"
        file_glob: .planning/*/task_plan.md

*, ? and [...] match within one directory level; a wildcard never crosses /, and * matches a leading dot. ** is refused. Matching is case-sensitive on every platform.
The pattern must match exactly one file. No match fails the check and lists the workspace; two or more fails it and names the matches. Neither is an error, and neither is a pass.
For a file that may be in one of several places, give a list. The check then needs exactly one file across all the entries together, and an entry may be a literal path:

    - kind: file-produced
      file_glob: [task_plan.md, .planning/*/task_plan.md]   # the root, or a named plan

A list needs at least two different entries (one pattern is written as a string), and every other rule here applies to each entry. A judge artifact takes a list the same way, - glob: [progress.md, .planning/*/progress.md], and is named in the prompt by its entries joined with |. - Only a regular file is a match, reached directly or through a symbolic link. A directory, a dangling link and a link to a FIFO are not, and a symbolic link to a directory is never entered. - A pattern with no wildcard, **, .., an absolute path, a backslash, a trailing / or /., an unclosed [, a [...] set that matches nothing (such as [z-a]; a set is read exactly as Python's fnmatch reads it), or file and file_glob on one assertion is an authoring error (exit 2). - Quote a pattern that starts with * or [: file_glob: "*.md". Unquoted, YAML reads * as an alias and [ as the start of a list, and the file does not load. - A pattern also matches the files the case seeds under workspace: files:. A content check may read a seeded file back, because the agent may have edited it, but a file-produced whose pattern matches a seeded file is an authoring error: it would pass whatever the skill did. So is one text used both as a path and as a glob in judge.artifacts. These rules belong to the case itself, so a case built in code gets them too. - The single match is read under the same rules as file:. A match that is a symbolic link pointing outside the workspace fails the check, with the refusal as evidence. - A product runner delivers the skill into the workspace (.claude/skills/<name>/, .agents/skills/<name>/, or skills_dir under cli), and leaves it there so --keep-workspace shows what the product saw. A file:, file_glob or judge artifact that reaches that copy would check harness files, not the skill's output, so the runner refuses it before any case runs (exit 2). */*/*/SKILL.md is refused this way.

Per-check results¶

assertions, trajectory and budget each report one result per declared check, not just one verdict for the whole block — the same shape the LLM judge already uses for its rubric. Every check carries a stable id, derived from what the case declared rather than from what the run produced, so the same id names the same check whether the skill was loaded or not. That is what makes comparative evals able to pair up a check across the candidate and baseline arms — including flagging one that passed either way as low-signal.

Source	Check id format	Example
`assertions`	`{kind}[{index}]` — positionally stable	`contains[0]`, `regex[2]`
`trajectory.called`	`called:{tool}`	`called:lookup_order`
`trajectory.forbidden`	`forbidden:{tool}`	`forbidden:issue_refund`
`trajectory.order`	`order`	`order`
`trajectory.max_calls`	`max_calls`	`max_calls`
`trajectory.skill_triggered`	`skill_triggered`	`skill_triggered`
`trajectory.call_args`	`call_args[{index}]` — positionally stable, so two entries may name one tool	`call_args[0]`
`budget.max_tokens`	`max_tokens`	`max_tokens`
`budget.max_cost_usd`	`max_cost_usd`	`max_cost_usd`
`budget.max_latency_ms`	`max_latency_ms`	`max_latency_ms`

Each check also carries evidence — the same text that would otherwise only appear in the evaluator's summary detail. In the JSON report, every score's checks list carries these ids and evidence regardless of whether a baseline ran; the console only prints evidence for checks that failed, to keep a passing run's output short.

The "a pass without evidence is a failure" rule (see Judging output quality above) is specific to the LLM judge, where an unsupported PASS is the characteristic failure mode it exists to defend against. It is not applied to assertion, trajectory or budget checks — their evidence is generated deterministically from the same comparison that produced the verdict, so it cannot go missing independently of it.

Reading `evals.json`¶

A skill that already has an evals/evals.json — the file the Agent Skills evaluation guide describes and Anthropic's skill-creator writes — runs with no YAML:

skill-lens run ./my-skill

{
  "skill_name": "csv-analyzer",
  "evals": [
    {
      "id": 1,
      "prompt": "Find the top 3 months by revenue in the CSV and chart them.",
      "expected_output": "A bar chart of the three highest months.",
      "files": ["evals/files/sales.csv"],
      "assertions": ["The chart shows exactly 3 months", "Both axes are labelled"]
    }
  ]
}

`evals.json`	Becomes	Rule
`skill_name`	—	Optional. When present it must equal the skill's `name`.
`id`	the case name `eval-<id>`	Required. An integer or a non-empty string, never a boolean, unique in the file. `eval-<id>` may not repeat the name of a case in the skill's other eval files either — see Names are unique.
`prompt`	`task`	Required, non-empty.
`assertions` or `expectations`	`judge.rubric`	A list of non-empty strings. The guide says `assertions`; `skill-creator` says `expectations`. Both in one eval is an error.
`expected_output`	`judge.expected`	Context for the judge. With no statements it becomes the single check `The output satisfies: <expected_output>`.
`files`	`workspace.files`	Paths relative to the skill directory, keyed in the workspace by the path as written.

What to know:

A real judge grades it. The statements are plain English, so judge = "fake" (the default) reports the cases as errored, never as passed. Pick a judge — see Judging output quality and Configuration.
Nothing to grade is an error. An eval with no statements and no expected_output stops the run (exit 2).
Keys are strict. An unknown key at the top level or in an eval stops the run (exit 2) and is named in the message. A file in another layout, such as one with a trigger block or files that are folder names, needs rewriting first.
files are text. Each is read when the file loads and must be a UTF-8 text file inside the skill directory, no larger than 1,000,000 bytes. A missing file, a folder, a binary file or a path that leaves the skill directory stops the run (exit 2). A case with no files gets no workspace.
A leading byte-order mark is skipped. Some Windows editors write one at the start of a JSON file. skill-lens ignores it, so the file loads as it would without it.
The judge reads the agent's reply. evals.json does not name output files, so a file the agent writes is not shown to the judge. Move to a YAML case with judge.artifacts when the skill's product is a file.
Upgrading can change a run. If a skill already had an evals/evals.json that earlier versions skipped, its cases now count toward the gate. Configure a judge, or they come back errored and the gate fails.
Move to YAML for more. Mock tools, trajectory, budget and mode: offered exist only in YAML. Both kinds of file can sit in one evals/ directory and are loaded together.

Where eval files are found¶

For each discovered skill, in order:

an evals/ directory beside SKILL.md — every .yaml / .yml file in it, plus evals.json (Reading evals.json), or
any *.eval.yaml file beside SKILL.md.

--evals <path> overrides discovery with an explicit file or directory. A file ending in .json is read as evals.json; in a directory, only a file named evals.json is read among the JSON files, so fixtures and schemas are left alone. Skills with no eval files are reported as skipped — visible in the output, never silently ignored.

A tool_libraries: entry resolves against the eval file's own directory whichever way the file was found, so --evals does not change where a library is looked for.

Names are unique¶

A case name is unique among one skill's cases, across every eval file found for it. A skill name is unique among the skills one run discovers. Both comparisons ignore letter case, so Greet and greet count as one name — the same way --case matches, which could never select one of the two without the other.

The reason is the report. It names each outcome by its skill, its case and its runner, so two cases sharing a name would be merged as if they were repetitions of one case. That gives wrong pass counts, a high-variance flag on two cases that never varied, a hidden low-signal flag, and two JUnit test cases nobody can tell apart. Two skills sharing a name would also share one per_skill_min rate.

A repeated name is an authoring error (exit 2), found when the files load, before any case runs; skill-lens list reports it too. The message names both places:

Two cases. Both files and both positions, even when they are in one file. A repeat can cross formats: a YAML case named eval-1 beside an evals.json with "id": 1 is one. Rename one of the two cases.
Two skills. Both directories. A skill with no name: takes its directory's name, so a/review/SKILL.md and b/review/SKILL.md with no name: repeat too. Rename one in its SKILL.md, or run skill-lens on each directory separately.