skill-lens¶
Run evaluations on Agent Skills (SKILL.md) — in CI/CD or on demand.
A skill is a directory containing a SKILL.md file. skill-lens discovers those
directories, finds the eval cases declared beside them, runs each case against a
runner, scores the result, and turns the whole run into a single exit code you can
gate a pipeline on.
Skills and their eval cases are inputs to the tool. Nothing about a skill under
test is vendored here, so any skill repository can adopt skill-lens without
embedding it.
Stability
This is 0.x. A minor release may still change behaviour, so pin what you depend on.
See the roadmap for what is shipped and what is planned.
Install¶
uv tool install "skill-lens[pydantic-ai]"
pip install "skill-lens[pydantic-ai]" works the same way. The pydantic-ai extra installs
the real-agent runner and judge; without it, skill-lens still runs the offline default
runner. The langchain extra installs the LangChain runner and judge the same way, and the
two can be installed together: "skill-lens[pydantic-ai,langchain]". From a checkout of this
repository, uv sync --extra pydantic-ai (or --extra langchain) installs the same thing,
and every command then runs as uv run skill-lens ....
An installed GitHub Copilot CLI or Claude Code needs no extra and no API key:
--runner copilot or --runner claude-code starts the product itself, and
judge = "copilot" or judge = "claude-code" in skill-lens.toml grades rubrics through
it — see Product runners.
To work on skill-lens itself, or to have the example skills to hand, install from a
checkout as described above. Make the checkout with:
git clone https://github.com/EmadMokhtar/skill-evaluator.git
cd skill-evaluator
Where to go next¶
| If you want to | Read |
|---|---|
| Write your first eval and run it | Getting started |
| Learn the vocabulary — case, runner, judge, arm, gate | Concepts |
| Look up an eval YAML field or assertion kind | Eval files |
| Look up a command or flag | CLI |
| Configure defaults for a repo | Configuration |
| Evaluate against a real agent, with tools and budgets | Runners |
| Understand exit codes and CI behavior | Gating |
| Look up an error message you are seeing | Troubleshooting |
| Understand how the tool is built | Architecture |
| Understand how a release is cut and published | Releasing |
| Work on skill-lens itself | Contributing |