Running skill-lens in CI

skill-lens is built to be a CI gate: the exit code is the contract — 0 gate passed, 1 gate failed, 2 a user or authoring error. Everything else on this page is about making that verdict legible.

Reports

Flag	Format	Read by
--json-output	JSON	Tooling, dashboards, artifact storage
--junit-output	JUnit XML	GitHub, GitLab, Jenkins, CircleCI, Buildkite test panes
--markdown-output	Markdown	$GITHUB_STEP_SUMMARY, PR comments

The JUnit mapping — including why only the candidate arm becomes test cases, and why an <error> can carry an evaluator's own diagnostic rather than the runner's — is documented in Gating and exit codes.

skill-lens never talks to the GitHub API. It renders a Markdown file; your workflow decides where that goes.

--markdown-max-chars only makes sense together with --markdown-output: it is rejected as a user error (exit 2) without it, and rejected outright below 1. Leave it unset for a step summary, which allows 1 MiB; set it near GitHub's 65,536-character comment cap when the same file will also be posted as a PR comment. Below the budget, truncation gives up detail before it gives up meaning — optional sections (totals, per-skill table, delta, failure detail) are dropped first, then gate reasons are elided behind a truthful +N more reasons count so a clipped comment never implies the reasons it shows were all of them. See --markdown-max-chars for the exact rule, including the one case (a budget too small to hold even the verdict) that falls back to a hard character cut.

The composite action

- uses: EmadMokhtar/skill-evaluator@v0.25.0
  with:
    path: ./skills
    runner: pydantic-ai
    model: openai:gpt-4o-mini
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Pin an exact tag, as shown above — until 1.0 a minor release may change behaviour, so there is no floating v0 tag to follow. Every release tags the tree in which install-spec already pins the matching version, so the action and the CLI it installs cannot drift apart.

To run against a commit that has not been released, pin the action to that commit SHA and give install-spec the same ref (skill-lens[pydantic-ai] @ git+https://github.com/EmadMokhtar/skill-evaluator@<commit-sha>), or reference the action as uses: ./ from a workflow inside this repository.

Every skill-lens run flag is available as a kebab-cased input (--min-pass-rate becomes min-pass-rate). runner takes one name — fake, pydantic-ai, langchain, copilot, claude-code or cli — or a comma-separated list to run every case through each (runner: pydantic-ai,langchain). Install every framework named: install-spec: skill-lens[pydantic-ai,langchain]==…; a product runner needs the product installed instead, see Running under a product. base-url points a keyed runner at a self-hosted endpoint the job can reach — a self-hosted GitHub runner beside the model server, typically — and is usually better committed as base_url in skill-lens.toml (see Self-hosted endpoints) than passed per job. Three more inputs are about the environment rather than the run:

Input	Default	Purpose
install-spec	skill-lens[pydantic-ai]==0.25.0	Passed verbatim to uv tool install. Accepts a PyPI name, a pinned version, a git ref, or a local path. Add the langchain extra (skill-lens[pydantic-ai,langchain], pinned the same way) when a job runs the LangChain runner.
working-directory	.	Directory to run in.
step-summary	true	Append the Markdown summary to $GITHUB_STEP_SUMMARY.

Outputs: exit-code, passed, pass-rate, json-report, junit-report, markdown-report.

allow-scripts: true mirrors keep-workspace: it passes --allow-scripts, so the skill's bundled scripts run on the job's runner. Enabling it in CI runs unvetted code with the job's filesystem readable. The script's environment is rebuilt from an allowlist, so GITHUB_TOKEN and every other secret in env are not inherited — but that is the only layer between a script and the token. actions/checkout persists the job token on disk by default (persist-credentials: true, in .git/config of the checkout), readable by the same user the script runs as; and a stock ubuntu-latest runner may not ship bwrap, or may refuse unprivileged user namespaces, in which case no OS sandbox applies (the report's scripts: on, sandbox: none (...) line says which), the network is open and nothing stops a script reading the token and sending it out. If you enable scripts, tell the checkout not to persist the token, and give the job only the permissions the run needs:

permissions:
  contents: read
steps:
  - uses: actions/checkout@v4
    with:
      persist-credentials: false
  - uses: EmadMokhtar/skill-evaluator@v0.25.0
    with:
      path: ./skills
      allow-scripts: true

Files on disk (the checkout, the runner's tools) stay readable either way. Turn scripts on only for skills you would run by hand. See Running bundled scripts.

The input being unset is not the same as false. Unset, the action lets skill-lens.toml decide — and in a pull_request workflow that file comes from the pull request, which can set allow_scripts = true and script_sandbox = "off" for itself. A fork's PR gets no secrets under plain pull_request, so a real runner fails at the missing key first; a pull_request_target workflow, a collaborator's PR, or a self-hosted runner has no such backstop. In those workflows pass allow-scripts: false explicitly, which overrides the file:

  - uses: EmadMokhtar/skill-evaluator@v0.25.0
    with:
      path: ./skills
      allow-scripts: false

json-output, junit-output and markdown-output default to real paths rather than being unset, because the action reads the JSON back to produce passed and pass-rate.

json-report, junit-report and markdown-report are always absolute paths, resolved against working-directory before they are written to $GITHUB_OUTPUT. A relative path is only meaningful inside the directory the run step used; a later step (upload-artifact, a custom script) has no reason to share that directory, so a relative output would be wrong whenever working-directory is not ..

The action runs its steps with shell: bash, which GitHub Actions executes under bash --noprofile --norc -eo pipefail — -e is already on before the action's own script runs a line. The run step captures the CLI's exit code explicitly (code=0; skill-lens run "${args[@]}" || code=$?) precisely so that -e cannot swallow a red gate before it is recorded, every reporting step after it carries if: always() so a failed run still gets its summary published and its outputs read, and the final step re-raises with exit "${CODE:-1}" — an empty code (the run step never completing at all: a failed install, a cancelled job) fails closed rather than defaulting to success. A gate that cannot prove it passed must fail.

The run step also deletes any report already sitting at json-output, junit-output and markdown-output before invoking the CLI. Exit code 2 means the CLI wrote nothing, so without that deletion a report left over from an earlier invocation in the same job — same working-directory, same default filenames — would be published and read as if it belonged to the run that just failed.

Two CI jobs guard that behavior end to end, beyond the unit test that only compares action.yml's inputs against the CLI's flags: action-smoke runs the action against a small passing fixture skill and asserts all three report files appear and are well-formed; action-smoke-failing-gate runs it against a fixture that deliberately fails and asserts the exit code is 1, passed is "false", and the reports still exist. The second job is the one that matters — a passing-only fixture would happily pass even if a future edit broke the exit-code capture (a bare failing command under -e, a step that silently swallows $?), so only a fixture that is supposed to go red can catch a regression in how red gets reported.

A complete workflow

# Gate pull requests on a skill-lens run, using the composite action.
#
# Copy this into .github/workflows/ in your own repository. Files under
# examples/ are inert -- only .github/workflows/ is executed by GitHub.
name: skill-lens

on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          # --baseline previous resolves the skill's prior version from git
          # history, so a shallow clone would leave it with nothing to compare.
          fetch-depth: 0

      - name: Evaluate the skills
        id: eval
        uses: EmadMokhtar/skill-evaluator@v0.25.0
        continue-on-error: true
        with:
          path: ./skills
          runner: pydantic-ai
          model: openai:gpt-4o-mini
          baseline: previous
          repeat: 3
          concurrency: 4
          # GitHub caps a comment at 65536 characters. Detail blocks are
          # dropped first, then gate reasons are elided behind a "+N more"
          # count; a budget too small for the verdict itself is cut outright.
          markdown-max-chars: "60000"
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Upload the reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: skill-lens-reports
          path: |
            ${{ steps.eval.outputs.json-report }}
            ${{ steps.eval.outputs.junit-report }}
            ${{ steps.eval.outputs.markdown-report }}

      - name: Comment on the pull request
        # GitHub withholds write tokens from fork-triggered `pull_request` runs,
        # so an unguarded comment step fails on exactly the PRs an open-source
        # project gets most. Fork PRs still get the step summary, the JUnit
        # rendering and the correct exit code. See docs/ci.md for the
        # `workflow_run` pattern if you need comments on forks too --
        # `pull_request_target` is not the answer.
        if: github.event.pull_request.head.repo.full_name == github.repository
        uses: actions/github-script@v7
        env:
          SUMMARY: ${{ steps.eval.outputs.markdown-report }}
        with:
          script: |
            const fs = require('fs');
            const summaryPath = process.env.SUMMARY;
            if (!summaryPath || !fs.existsSync(summaryPath)) {
              core.info('No Markdown summary was written; skipping the comment.');
              return;
            }
            const marker = '<!-- skill-lens -->';
            const body = marker + '\n' + fs.readFileSync(summaryPath, 'utf8');
            const comments = await github.paginate(github.rest.issues.listComments, {
              ...context.repo,
              issue_number: context.issue.number,
              per_page: 100,
            });
            const existing = comments.find(
              (c) => c.user.type === 'Bot' && c.body.includes(marker)
            );
            if (existing) {
              await github.rest.issues.updateComment({
                ...context.repo, comment_id: existing.id, body,
              });
            } else {
              await github.rest.issues.createComment({
                ...context.repo, issue_number: context.issue.number, body,
              });
            }

      - name: Fail the build if the gate failed
        # `continue-on-error` above let the comment and artifact steps run, so
        # the gate's verdict is re-raised here. An empty exit code means the
        # evaluation never completed, which must fail rather than pass.
        if: always() && steps.eval.outputs.exit-code != '0'
        env:
          CODE: ${{ steps.eval.outputs.exit-code }}
        run: exit "${CODE:-1}"

Gating on the delta

The complete workflow above is already comparative: baseline: previous re-runs every case against the skill's previous version, and repeat: 3 samples each arm three times. Both are ordinary inputs, and min-delta is the third — it turns the delta the report carries into a verdict:

      - uses: EmadMokhtar/skill-evaluator@v0.25.0
        with:
          path: ./skills
          runner: pydantic-ai
          model: openai:gpt-4o-mini
          baseline: previous      # none, or previous resolved from git
          repeat: 3               # sample each arm three times
          min-delta: "0.0"        # must not regress
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

baseline: previous reads git history, so the checkout must be unshallow — fetch-depth: 0, as in the workflow above. min-delta without baseline is a user error (exit 2), and a delta gate with nothing comparable fails rather than passing vacuously: the rules are in Comparative evals. Two arms sampled three times is six runs per case — see Concurrency and cost.

Running the matrix

One job, one report, every case through both frameworks:

      - uses: EmadMokhtar/skill-evaluator@v0.25.0
        with:
          path: ./skills
          install-spec: "skill-lens[pydantic-ai,langchain]==0.25.0"
          runner: pydantic-ai,langchain
          model: openai:gpt-4o-mini
          markdown-output: skill-lens.md
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

A case failing under either framework fails the job, and the summary names the runner on every failing line. A two-runner job spends twice.

The alternative is a GitHub strategy.matrix over runner: [pydantic-ai, langchain], one job per framework with the single-name runner: input. It needs no list support and shows one check per framework in the pull request, at the cost of one summary and one JUnit file per job rather than one for the whole matrix.

Running under a product

A product runner needs the product installed on the runner and its token in the environment; no provider API key and no framework extra. Pin runner: so a pull request cannot pick the product for itself:

- uses: actions/setup-node@v4
  with: { node-version: 22 }
- run: npm install -g @github/copilot
- uses: EmadMokhtar/skill-evaluator@v0.25.0
  with:
    path: ./skills
    runner: copilot
  env:
    COPILOT_GITHUB_TOKEN: ${{ secrets.COPILOT_TOKEN }}

Each case spends the product's quota; the Plan: line prints the ceiling. The product runs with its permission prompts disabled, and it holds its token and whatever it can read on the runner (the product sandbox narrows what it inherits, not what it can read), so give the token only to jobs whose checkout you trust — see Security for why pinning runner: alone is not enough. The model input is refused (exit 2) when only product runners are named and no keyed judge falls back to it, and so is judge-model when the file's judge is a product (copilot, claude-code or cli — see Judging with a product); set the product's model with [runners.<name>] args in skill-lens.toml, which serves the runner and the judge alike.

The product runs under the product sandbox: by default a temporary home directory and an allowlisted environment, so it signs in from its token (COPILOT_GITHUB_TOKEN, CLAUDE_CODE_OAUTH_TOKEN), never from a login on the runner, and it inherits only the allowlist: its own variables (Copilot just the first of COPILOT_GITHUB_TOKEN, GH_TOKEN, GITHUB_TOKEN that is set), pass_env, and the plain ones such as PATH. It can still read the job's other variables the way any same-user process can, which Security explains. On a stock ubuntu-latest runner bwrap may be missing, or unprivileged user namespaces refused, and then no OS sandbox applies; the product line on the report says which. The product-sandbox input passes --product-sandbox. Unset, skill-lens.toml decides — and in a pull_request workflow that file comes from the pull request, which can set product_sandbox = "off" for itself. Where the checkout is a pull request you do not trust, pass required, so the run stops (exit 2) rather than running without a sandbox:

  - uses: EmadMokhtar/skill-evaluator@v0.25.0
    with:
      path: ./skills
      runner: claude-code
      product-sandbox: required
    env:
      CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}

Without the action

# The same gate without the composite action, for repositories that would
# rather call the CLI directly.
#
# Copy this into .github/workflows/ in your own repository.
name: skill-lens (CLI)

on:
  pull_request:

permissions:
  contents: read

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install uv
        uses: astral-sh/setup-uv@v5
        with:
          enable-cache: true

      - name: Install skill-lens
        run: uv tool install "skill-lens[pydantic-ai]"

      - name: Evaluate the skills
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          skill-lens run ./skills \
            --runner pydantic-ai \
            --model openai:gpt-4o-mini \
            --concurrency 4 \
            --json-output skill-lens-report.json \
            --junit-output skill-lens-junit.xml \
            --markdown-output skill-lens-summary.md

      - name: Publish the summary
        # `always()` so a failing gate still reports why.
        if: always()
        run: cat skill-lens-summary.md >> "$GITHUB_STEP_SUMMARY"

Pull request comments on forks

GitHub deliberately gives pull_request runs triggered from a fork a read-only token, so a comment step fails there no matter what permissions: says. The example above guards the comment step with:

if: github.event.pull_request.head.repo.full_name == github.repository

Fork PRs still get the step summary, the JUnit rendering and the correct exit code — the substance of the report. Only the comment is skipped.

If you need comments on fork PRs, use the two-workflow workflow_run pattern: the pull_request workflow runs the evaluation and uploads the Markdown as an artifact, never holding a write token; a second workflow triggered on workflow_run downloads it and posts the comment with pull-requests: write.

Do not reach for pull_request_target instead. It runs the base repository's workflow with a write token and access to secrets, and checking out the pull request's head under it is one of the best-known ways to hand a fork's code your repository's credentials.

Concurrency and cost

--concurrency N runs N cases at once. The work is network-bound — one provider round trip per case against sub-millisecond of local work — so this overlaps waiting rather than using more cores, and the practical ceiling is your provider's rate limit.

It does not change what a run costs. --baseline and --repeat do: --baseline previous --repeat 3 is six runs per case, not one. skill-lens run prints a ceiling estimate before it starts whenever the runner needs an API key.

To cap what a job spends, set max-spend (or commit max_spend_usd): once the run has recorded that many US dollars, no new case starts, the cases left are reported as not run, and the gate fails. Cases already running when it is crossed finish, so with concurrency: 4 a job can end above the ceiling by what four cases spend. See The spend ceiling.

Discovery (walking skills, loading eval files, filtering by --tag) always finishes, for every skill, before any case runs — independent of --concurrency. A malformed eval file anywhere therefore aborts the whole run before a single provider call is made, rather than after some other skill's cases already ran and were paid for.