Comparative evals¶

Why¶

A case that only ever runs with the skill loaded can tell you the score, but not what earned it. If the model would have answered the same way with no skill at all, a green case is measuring the model, not the skill. Comparative evals close that gap by running each case twice — once with the skill, once without (or against an earlier version of it) — and reporting the difference.

The two arms¶

Every case can run in two arms:

candidate — the skill under test, exactly as it runs today. This is the only arm a run without --baseline produces, and the only arm min_pass_rate, per_skill_min, fail_on_error and the zero-cases check ever read. --min-delta is the one gate rule that reads both, because the delta it gates on is built from both.
baseline — a comparison point, selected with --baseline:
none — an empty skill: same name, no description, no instructions. This isolates what the skill's text contributes, as opposed to what the model would do unprompted.
previous — the skill's prior version, resolved from git (see below). This isolates what a specific edit changed.

Omitting --baseline is what turns comparison off. none names a kind of baseline (an empty skill), not the absence of one — so leaving the flag unset, not passing --baseline none, is the only way to get a single-arm run. With no flag, skill-lens run keeps the single-arm layout: one arm, one line per outcome, no delta block. It is not byte-identical, though — a failing case's assertion, trajectory and budget evaluators now emit per-check evidence (once, only the judge evaluator did this), so a failing outcome prints one indented line per failed check where it printed none before. That is strictly more information, not a behavior change in what runs. Upgrading to a version of skill-lens that supports comparison must never silently double anyone's bill.

The baseline skill is built once per skill, per run — not once per case, and not once per repetition:

`--baseline`	Baseline skill
`none`	`name` kept (for grouping in the report); `description` and `instructions` both empty
`previous`	The prior version parsed from git, same `name` and `path` as recorded in that commit

The baseline arm never receives the skill's name, description, or instructions when both are empty. The system prompt normally opens with # {skill.name} as a header; for an empty --baseline none skill that would leak the skill's own name into the very prompt used to measure whether the skill's text mattered. Instead, a skill whose description and instructions are both empty gets a neutral preamble instead of that header:

You are a helpful assistant.

The rule keys on emptiness, not on which arm is running, so no runner has to know — or be trusted to know — which arm it is serving.

A case's mock tools (tools:) are unaffected by the arm. They are the environment the case declares, not part of the skill, so both arms see the same tools and the comparison stays honest.

flowchart TD
    CASE["One eval case"] --> CAND["Candidate arm: SKILL.md as it is now"]
    CASE --> BASE["Baseline arm: --baseline none or previous"]
    CAND --> CR["Run it --repeat N times"]
    BASE --> BR["Run it --repeat N times"]
    CR --> PAIR{"Can both arms be honestly compared?"}
    BR --> PAIR
    PAIR -->|"no: skipped, unresolvable, or every repetition errored"| DROP["Excluded from BOTH halves of the delta"]
    PAIR -->|yes| DELTA["Delta: pass rate, tokens, cost, latency"]
    CR --> GATE["The gate: candidate outcomes, plus the delta under --min-delta"]
    DELTA -->|"--min-delta, when set"| GATE

How `previous` is resolved¶

--baseline previous walks the skill's own git history, rooted at its directory:

Confirm the directory is inside a git repository.
Confirm SKILL.md is tracked.
List the commits that touched SKILL.md at its current path, newest first. Where one of them created that path by moving SKILL.md there byte for byte, carry on from the old path, before the move (see When a skill has moved). The last 50 commits are read in total, across every path.
Read each candidate commit's SKILL.md and parse it. The first one that qualifies as genuinely earlier wins:
if the working copy declares a version:, the first commit whose version differs from the working copy's;
if it declares none, the first commit whose content differs from the working copy's.
Extract the files the skill shipped at that same commit, by the same rule as the candidate's bundle, from the directory the skill had at that commit (git archive, then tarfile with its data filter) into a temporary directory that is deleted when the run ends, so the previous instructions are paired with the previous bundle — never with the candidate's scripts. A commit that shipped no such file gives the baseline no bundle tools. See Bundled files and scripts.

A declared version: is the stronger signal — an edit that did not bump it is still considered this version, however much prose changed underneath it. Without a declared version, differing content is the best evidence available. The version is also the authority for the bundle: it comes from the commit that last edited SKILL.md at the previous version, so a commit that changed only scripts/ after that edit is invisible to the baseline.

The comparison is against the working copy, not HEAD — so uncommitted edits to SKILL.md are what run as the candidate. This matters for local iteration: you do not need to commit a change before measuring it.

When a skill has moved¶

Git records a move as a delete at the old path and an add at the new one, so the history of the new path begins at the move. The resolver carries on from the old path only when the commit that created the new path moved SKILL.md there unchanged — the git mv of a skill's directory, committed on its own. That is certain to be the same file.

Two things look like a move and are not followed:

A copy, even one whose source is deleted in a later commit. A skill started from another skill's SKILL.md must never be compared against that skill's old instructions. This is why skill-lens does not use git log --follow: it follows copies as well as renames, and matches by similarity.
A move that also edited SKILL.md in the same commit. It may well be the same skill, but nothing can prove it, and a baseline taken from another file would be worse than none.

Either one ends the search with no earlier version found, which the report explains. To keep a skill's history across a move, commit the move by itself, then edit.

Resolution happens once per skill, not per case or per repetition — it shells out to git, and shelling out once per repetition would multiply subprocess calls by nothing useful.

When resolution fails¶

Resolution can fail for reasons that are facts about the repository, not about the skill's content. Each is reported as its own message, and none of them raise — the same discipline runners follow for provider failures:

Reason	Meaning
`git` is not installed	The environment cannot answer the question at all
not a git repository	The skill's directory (or an ancestor) has no `.git`
`SKILL.md` is not tracked by git	The file exists but was never committed
cannot read the working copy's `SKILL.md`	Filesystem error (file missing, permission denied) or character encoding issue
no earlier version found within the searched history	Every commit in the last 50 has the same version (or, unversioned, the same content), or the history begins at a copy, or at a move that also edited `SKILL.md` — see When a skill has moved
cannot archive commit `<sha>`	`git archive` failed or exceeded the 10-second timeout — a very large historical `assets/` can do that
cannot extract the bundle at commit `<sha>`	The archive would not parse, or a member failed the safe-extraction filter, or the extraction hit a filesystem error; a half-extracted directory is removed

An unresolvable baseline is reported, never assumed to be "no change". Treating "we couldn't check" as "nothing changed" would let a repository pass --min-delta forever by deleting its git history. Without --min-delta, an unresolved baseline is just a note on the report. With it, it fails the gate (see --min-delta below).

`mode: offered` cases¶

mode: offered cases (see Did the agent reach for the skill?) measure whether the agent chose to trigger the skill at all, which makes them the highest- value case for comparison: a description edit is exactly what a trigger-rate delta measures.

`--baseline`	Behavior
`previous`	Both arms run. The baseline offers the skill's old name and description; the difference in trigger rate between the two arms is how a description edit is measured.
`none`	Candidate-only. There is no skill to offer in an empty baseline, so `skill_triggered` would be `false` by construction. Running it anyway would spend real money to prove a tautology, and would report the structural artifact as "the skill helped 100%".

A skipped baseline (offered case, --baseline none) is recorded as a note on the report and excludes that case from the delta. It is not a baseline failure — nothing went wrong, the comparison just isn't meaningful for that case.

`--repeat N`¶

--repeat N samples each arm N times per case. Each repetition is its own outcome — it is scored independently and contributes its own pass/fail to the report, rather than being collapsed into one verdict per case.

At the default min_pass_rate = 1.0, this is exactly "all repetitions must pass": a single failing repetition drops the pass rate below 1.0 and fails the gate, the same as it would for one unrepeated case. Below 1.0, a pass rate degrades proportionally — 4 of 5 repetitions passing is a different signal from 0 of 5, and repeats let you see that instead of averaging it away.

Repeats also feed high-variance detection: repetitions that disagree with each other are flagged, which is only meaningful once there is more than one to compare.

The delta block¶

skill-lens compares the two arms and reports the difference — the delta. Every delta is candidate minus baseline:

Metric	Sign convention
Pass rate	Higher is better (a positive delta means the candidate scored better)
Tokens, cost, latency	Negative is better (a positive delta means the candidate cost or took more)

The reporters say which direction is better next to each number rather than leaving a reader to work it out.

The delta is paired. A case is only comparable when both arms produced at least one non-errored repetition. Anything that breaks the pair — a skipped baseline (offered + none), an unresolvable baseline, or every repetition of one arm erroring — excludes the case from both halves of the delta, with the reason recorded. Keeping the candidate half of a broken pair would silently bias the aggregate number, so it is dropped along with its partner. Aggregate rates are computed only over comparable cases' outcomes.

Console output collapses to one line per case — both arms summarized together — rather than one line per outcome, since --repeat 5 --baseline previous would otherwise print ten lines for a single case:

[PASS] order-support :: refund-request (pydantic-ai)  candidate 5/5  baseline 2/5  +60%

followed by a delta block naming the arms it compared:

Delta vs baseline (previous)
  pass rate  40% -> 100%  +60%   (higher is better)
  tokens     -60   (negative is better)
  cost       $-0.0002   (negative is better)
  latency    -120ms   (negative is better)

With more than one runner, the delta is reported per runner. Runners spend very differently — a product runner can use about 375,000 tokens on a case a framework runner does in 9,000 — so one number averaged across them describes neither, and a runner that got better can hide one that got worse. The block then has one section per runner, in the order each first ran, and a runner with no comparable case says so:

Delta vs baseline (none), per runner
  claude-code
    pass rate  0% -> 60%  +60%   (higher is better)
    tokens     +301804   (negative is better)
    cost       $+0.1900   (negative is better)
    latency    +24000ms   (negative is better)
  pydantic-ai
    pass rate  0% -> 100%  +100%   (higher is better)
    tokens     +240000   (negative is better)
    cost       $+0.0100   (negative is better)
    latency    +149000ms   (negative is better)

A runner none of whose cases had both arms gets one line instead of four:

  pydantic-ai: no case had a comparable baseline

The Markdown report shows the same as a table with one row per runner. The JSON report keeps the aggregate numbers at the top of delta and adds delta.runners, one entry per runner with the same fields plus comparable_cases. A single-runner run prints exactly the block above.

Low-signal checks and high-variance cases¶

Two diagnostics ride along with the delta. Both describe the eval suite, not the skill under test, and neither changes the exit code.

Low-signal check — a check id that passed in every scored repetition of both arms of a comparable case. It contributed to the with-skill score, but it didn't measure anything about the skill, since it also held with no skill loaded at all. It is a candidate for tightening or removing.
High-variance case — a (case, arm) whose repetitions were not unanimous — some passed, some didn't. Only meaningful with --repeat > 1. An unstable pass rate for the same skill, same case, same runner usually points at ambiguous instructions in the skill rather than at a flaky provider (though a genuinely flaky provider looks identical, which is exactly why this is advisory and not a gate).

Each flag line names the runner, the way a case line does, because under two runners the same case and check can be flagged once per runner:

Low-signal checks (passed with and without the skill — they measure nothing):
  - order-support :: refund-request (claude-code): contains[0]
  - order-support :: refund-request (pydantic-ai): contains[0]

High-variance cases (repetitions disagreed — often ambiguous instructions):
  - order-support :: refund-request (pydantic-ai, candidate): 60%, stddev 0.49

In the JSON report, every delta.low_signal[] and delta.high_variance[] entry carries a runner field; the high-variance entries always had one, and the low-signal entries now do.

Both are reported for visibility so a suite's weak spots are named, but neither ever fails the gate. A flag that could block a merge for "this assertion is weak" trains people to ignore flags, and there is no way to tell a bad skill apart from a flaky provider from the flag alone.

`--min-delta`¶

--min-delta <float> turns the delta into a gate: the candidate's pass-rate delta must be at least this much better than the baseline's.

--min-delta requires a baseline. Passing it without --baseline is a user error and exits 2 — the alternative is a gate that silently checks nothing, which is the same vacuous-pass failure mode every other gate rule in this project rejects.

With a baseline set, --min-delta fails the gate for four reasons, checked in this order:

No baseline arm ran at all — every case's baseline was skipped, so there is nothing to build a delta from in the first place. A suite made entirely of skipped-baseline offered cases under --baseline none fails through this rule, not the next one: those cases produce no baseline outcomes at all, so there is no delta to look inside.
No case was comparable — mirroring "a run executing zero cases fails the gate": a gate that verified nothing must never report a pass. This is the rule for a delta that was built and then emptied — some baseline did run, but every pair was dropped.
The delta is below the bar — pass_rate_delta < min_delta. With more than one runner this is checked per runner: every runner's pass-rate delta must meet the bar, and a runner with no comparable case fails with its own reason, no case under runner '<name>' had a comparable baseline. The aggregate across runners is never gated on, because a runner that improved could carry one that regressed.
A skill's baseline could not be resolved — naming the skill and the reason. Otherwise a repository could pass --min-delta forever by deleting its git history.

A deliberately skipped baseline (an offered case under --baseline none) is, on its own, not a gate reason — nothing went wrong. It becomes one only indirectly: through rule 1 if every case skips its baseline, or through rule 2 if the pairs that do run leave nothing comparable behind.

Baseline outcomes count toward --min-delta and no other gate rule. Every other gate rule — min_pass_rate, per_skill_min, fail_on_error, the zero-cases check — reads the candidate arm only. A strong baseline means the skill was unnecessary, not that CI should go red; the baseline exists to be compared against, not to be graded on its own.

Cost¶

Comparison multiplies spend. --repeat 5 --baseline previous runs 10x as many cases as a plain run: 5 repetitions x 2 arms. Before starting, when the selected runner needs an API key, the CLI prints a run plan:

Plan: up to 2 arm(s) x 3 repeat(s) x 1 runner(s) x 4 case(s) = 24 runs

This is deliberately a ceiling, not a forecast — "up to", not "exactly". It applies the --tag filter (which only ever reduces the count), but it does not resolve baselines or evaluate per-case arm rules (mode: offered under --baseline none drops the baseline arm for that case; an unresolvable baseline drops it for the whole skill). Reproducing those here would mean duplicating the orchestrator's discovery — including its git calls — just to print a line, and both of those only ever bring the real total down from the number shown.

`version:` and why it must be quoted¶

A skill's frontmatter can declare version:, which --baseline previous uses to find the prior release (see above). It must parse as a YAML string:

---
name: order-support
version: "1.2.0"
---

An unquoted decimal like version: 1.20 is rejected at parse time as an authoring error (SkillParseError, exit 2) rather than silently accepted. YAML resolves 1.20 and 1.2 to the same floating-point value, so two genuinely different versions written that way would silently compare equal — and --baseline previous would report zero change between versions that actually differ. Three-part semantic versions like 1.0.0 are already unambiguous strings in YAML and need no quoting, but quoting is always safe and recommended.

A skill with no version: at all is not an error — resolution falls back to comparing file content instead (see above).

Try it on the shipped examples¶

examples/greeting is versioned for exactly this. Version 1.1.0 added one instruction — "do not use exclamation marks" — and one assertion only that version satisfies (not_contains: "!"). Because 1.0.0 is in this repository's history, a checkout can run the comparison as-is:

skill-lens run examples/greeting --runner pydantic-ai --baseline previous

The candidate arm runs the working copy; the baseline arm runs 1.0.0 resolved from git. Version 1.0.0 typically answers Hello, Ada!, so the baseline fails the new assertion, the candidate passes it, and the delta block reports the improvement — a small, real example of what --min-delta gates on.

Worked CI example¶

skill-lens run ./skills --runner pydantic-ai --baseline previous --repeat 3 --min-delta 0.0

This runs every case three times in each arm, compares the candidate's current SKILL.md against the newest earlier version git can find, and fails the build if the candidate did not do at least as well as that earlier version (--min-delta 0.0 — "must not regress"). It also fails if a skill's history has no earlier version to compare against, or if nothing in the suite was comparable at all.