Check whether the instructions changed the result
agent-skills-eval runs each evaluation twice: once with the skill loaded and once without it. A judge model grades both outputs against the same assertions, then the CLI writes JSON artefacts and a static side-by-side HTML report.
Point it at a small skill with two or three cases before attempting a large
suite. Write assertions for behaviour you can recognise, run with --baseline,
and inspect the failures rather than relying only on the rolled-up score. Swipe
Radar verified that the published CLI exposes the documented baseline, target,
judge, strict-validation, workspace, and report options.
We also connected the SDK to a local Codex provider and evaluated the Swipe
Radar skill with and without its instructions. The run wrote separate output
and grading artefacts, a rolled-up benchmark, and a static HTML report. That
verified the custom-provider and baseline paths against a real skill rather
than stopping at --help.
A comparison is evidence, not ground truth
The target and judge both consume model calls, so a real run has a cost. A model judge can also prefer style or wording that does not reflect actual task success. Use deterministic assertions for file shape, tool calls, and exact requirements where possible, then review a sample of the model-graded cases yourself.
Custom providers also own their usage accounting. Our local Codex adapter did not return token or cost data, so those report fields stayed at zero even though the model calls ran. Do not compare cost or token deltas until the provider reports those values.