Running skill-lens in CI¶

skill-lens is built to be a CI gate: the exit code is the contract — 0 gate passed, 1 gate failed, 2 a user or authoring error. Everything else on this page is about making that verdict legible.

Reports¶

Flag	Format	Read by
`--json-output`	JSON	Tooling, dashboards, artifact storage
`--junit-output`	JUnit XML	GitHub, GitLab, Jenkins, CircleCI, Buildkite test panes
`--markdown-output`	Markdown	`$GITHUB_STEP_SUMMARY`, PR comments

The JUnit mapping — including why only the candidate arm becomes test cases, and why an <error> can carry an evaluator's own diagnostic rather than the runner's — is documented in Gating and exit codes.

skill-lens never talks to the GitHub API. It renders a Markdown file; your workflow decides where that goes.

--markdown-max-chars only makes sense together with --markdown-output: it is rejected as a user error (exit 2) without it, and rejected outright below 1. Leave it unset for a step summary, which allows 1 MiB; set it near GitHub's 65,536-character comment cap when the same file will also be posted as a PR comment. Below the budget, truncation gives up detail before it gives up meaning — optional sections (totals, per-skill table, delta, failure detail) are dropped first, then gate reasons are elided behind a truthful +N more reasons count so a clipped comment never implies the reasons it shows were all of them. See --markdown-max-chars for the exact rule, including the one case (a budget too small to hold even the verdict) that falls back to a hard character cut.

The composite action¶

- uses: EmadMokhtar/skill-evaluator@v0.2.0
  with:
    path: ./skills
    runner: pydantic-ai
    model: openai:gpt-4o-mini
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Pin an exact tag, as shown above — until 1.0 a minor release may change behaviour, so there is no floating v0 tag to follow. Every release tags the tree in which install-spec already pins the matching version, so the action and the CLI it installs cannot drift apart.

To run against a commit that has not been released, pin the action to that commit SHA and give install-spec the same ref (skill-lens[pydantic-ai] @ git+https://github.com/EmadMokhtar/skill-evaluator@<commit-sha>), or reference the action as uses: ./ from a workflow inside this repository.

Every skill-lens run flag is available as a kebab-cased input (--min-pass-rate becomes min-pass-rate), plus three inputs about the environment rather than the run:

Input	Default	Purpose
`install-spec`	`skill-lens[pydantic-ai]==0.2.0`	Passed verbatim to `uv tool install`. Accepts a PyPI name, a pinned version, a git ref, or a local path.
`working-directory`	`.`	Directory to run in.
`step-summary`	`true`	Append the Markdown summary to `$GITHUB_STEP_SUMMARY`.

Outputs: exit-code, passed, pass-rate, json-report, junit-report, markdown-report.

json-output, junit-output and markdown-output default to real paths rather than being unset, because the action reads the JSON back to produce passed and pass-rate.

json-report, junit-report and markdown-report are always absolute paths, resolved against working-directory before they are written to $GITHUB_OUTPUT. A relative path is only meaningful inside the directory the run step used; a later step (upload-artifact, a custom script) has no reason to share that directory, so a relative output would be wrong whenever working-directory is not ..

The action runs its steps with shell: bash, which GitHub Actions executes under bash --noprofile --norc -eo pipefail — -e is already on before the action's own script runs a line. The run step captures the CLI's exit code explicitly (code=0; skill-lens run "${args[@]}" || code=$?) precisely so that -e cannot swallow a red gate before it is recorded, every reporting step after it carries if: always() so a failed run still gets its summary published and its outputs read, and the final step re-raises with exit "${CODE:-1}" — an empty code (the run step never completing at all: a failed install, a cancelled job) fails closed rather than defaulting to success. A gate that cannot prove it passed must fail.

The run step also deletes any report already sitting at json-output, junit-output and markdown-output before invoking the CLI. Exit code 2 means the CLI wrote nothing, so without that deletion a report left over from an earlier invocation in the same job — same working-directory, same default filenames — would be published and read as if it belonged to the run that just failed.

Two CI jobs guard that behavior end to end, beyond the unit test that only compares action.yml's inputs against the CLI's flags: action-smoke runs the action against a small passing fixture skill and asserts all three report files appear and are well-formed; action-smoke-failing-gate runs it against a fixture that deliberately fails and asserts the exit code is 1, passed is "false", and the reports still exist. The second job is the one that matters — a passing-only fixture would happily pass even if a future edit broke the exit-code capture (a bare failing command under -e, a step that silently swallows $?), so only a fixture that is supposed to go red can catch a regression in how red gets reported.

A complete workflow¶

# Gate pull requests on a skill-lens run, using the composite action.
#
# Copy this into .github/workflows/ in your own repository. Files under
# examples/ are inert -- only .github/workflows/ is executed by GitHub.
name: skill-lens

on:
  pull_request:

permissions:
  contents: read
  pull-requests: write

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          # --baseline previous resolves the skill's prior version from git
          # history, so a shallow clone would leave it with nothing to compare.
          fetch-depth: 0

      - name: Evaluate the skills
        id: eval
        uses: EmadMokhtar/skill-evaluator@v0.2.0
        continue-on-error: true
        with:
          path: ./skills
          runner: pydantic-ai
          model: openai:gpt-4o-mini
          baseline: previous
          repeat: 3
          concurrency: 4
          # GitHub caps a comment at 65536 characters. Detail blocks are
          # dropped first, then gate reasons are elided behind a "+N more"
          # count; a budget too small for the verdict itself is cut outright.
          markdown-max-chars: "60000"
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

      - name: Upload the reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: skill-lens-reports
          path: |
            ${{ steps.eval.outputs.json-report }}
            ${{ steps.eval.outputs.junit-report }}
            ${{ steps.eval.outputs.markdown-report }}

      - name: Comment on the pull request
        # GitHub withholds write tokens from fork-triggered `pull_request` runs,
        # so an unguarded comment step fails on exactly the PRs an open-source
        # project gets most. Fork PRs still get the step summary, the JUnit
        # rendering and the correct exit code. See docs/ci.md for the
        # `workflow_run` pattern if you need comments on forks too --
        # `pull_request_target` is not the answer.
        if: github.event.pull_request.head.repo.full_name == github.repository
        uses: actions/github-script@v7
        env:
          SUMMARY: ${{ steps.eval.outputs.markdown-report }}
        with:
          script: |
            const fs = require('fs');
            const summaryPath = process.env.SUMMARY;
            if (!summaryPath || !fs.existsSync(summaryPath)) {
              core.info('No Markdown summary was written; skipping the comment.');
              return;
            }
            const marker = '<!-- skill-lens -->';
            const body = marker + '\n' + fs.readFileSync(summaryPath, 'utf8');
            const comments = await github.paginate(github.rest.issues.listComments, {
              ...context.repo,
              issue_number: context.issue.number,
              per_page: 100,
            });
            const existing = comments.find(
              (c) => c.user.type === 'Bot' && c.body.includes(marker)
            );
            if (existing) {
              await github.rest.issues.updateComment({
                ...context.repo, comment_id: existing.id, body,
              });
            } else {
              await github.rest.issues.createComment({
                ...context.repo, issue_number: context.issue.number, body,
              });
            }

      - name: Fail the build if the gate failed
        # `continue-on-error` above let the comment and artifact steps run, so
        # the gate's verdict is re-raised here. An empty exit code means the
        # evaluation never completed, which must fail rather than pass.
        if: always() && steps.eval.outputs.exit-code != '0'
        env:
          CODE: ${{ steps.eval.outputs.exit-code }}
        run: exit "${CODE:-1}"

Without the action¶

# The same gate without the composite action, for repositories that would
# rather call the CLI directly.
#
# Copy this into .github/workflows/ in your own repository.
name: skill-lens (CLI)

on:
  pull_request:

permissions:
  contents: read

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Install uv
        uses: astral-sh/setup-uv@v5
        with:
          enable-cache: true

      - name: Install skill-lens
        run: uv tool install "skill-lens[pydantic-ai]"

      - name: Evaluate the skills
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          skill-lens run ./skills \
            --runner pydantic-ai \
            --model openai:gpt-4o-mini \
            --concurrency 4 \
            --json-output skill-lens-report.json \
            --junit-output skill-lens-junit.xml \
            --markdown-output skill-lens-summary.md

      - name: Publish the summary
        # `always()` so a failing gate still reports why.
        if: always()
        run: cat skill-lens-summary.md >> "$GITHUB_STEP_SUMMARY"

Pull request comments on forks¶

GitHub deliberately gives pull_request runs triggered from a fork a read-only token, so a comment step fails there no matter what permissions: says. The example above guards the comment step with:

if: github.event.pull_request.head.repo.full_name == github.repository

Fork PRs still get the step summary, the JUnit rendering and the correct exit code — the substance of the report. Only the comment is skipped.

If you need comments on fork PRs, use the two-workflow workflow_run pattern: the pull_request workflow runs the evaluation and uploads the Markdown as an artifact, never holding a write token; a second workflow triggered on workflow_run downloads it and posts the comment with pull-requests: write.

Do not reach for pull_request_target instead. It runs the base repository's workflow with a write token and access to secrets, and checking out the pull request's head under it is one of the best-known ways to hand a fork's code your repository's credentials.

Concurrency and cost¶

--concurrency N runs N cases at once. The work is network-bound — one provider round trip per case against sub-millisecond of local work — so this overlaps waiting rather than using more cores, and the practical ceiling is your provider's rate limit.

It does not change what a run costs. --baseline and --repeat do: --baseline previous --repeat 3 is six runs per case, not one. skill-lens run prints a ceiling estimate before it starts whenever the runner needs an API key.

Discovery (walking skills, loading eval files, filtering by --tag) always finishes, for every skill, before any case runs — independent of --concurrency. A malformed eval file anywhere therefore aborts the whole run before a single provider call is made, rather than after some other skill's cases already ran and were paid for.