Skip to content

CLI Reference ​

skill-up provides the following commands, covering the full evaluation lifecycle: validate → run → list cases → generate reports → import legacy formats.


skill-up run ​

Run evaluation cases and produce reports.

bash
skill-up run [path] [flags]

Argument ​

ArgumentDescription
pathPath to eval.yaml. Defaults to evals/eval.yaml in the current dir

Flags ​

FlagDefaultDescription
--autofalseAuto-detect the evals/ directory; can directly consume an Anthropic evals.json
--include-case-name—Run only matching cases (glob; can be repeated)
--exclude-case-name—Exclude matching cases (glob; can be repeated)
--format—Extra report formats: junit / html (repeatable). result.json is always written; --format junit produces report.xml, --format html produces report.html; --format json is a no-op
--output-dir<skill-name>-workspace/ next to the skill dirOutput directory for reports and artifacts
--workspace—Reuse an existing host directory as the agent workspace. Supported only with environment.type: none, cases.parallelism: 1, and benchmark disabled. The agent can modify this directory, but skill-up never deletes it.
--iteration0 (auto)Repeat selected cases for stability/flakiness sampling. 0 auto-appends one run after the latest iteration-N/ without summarizing history; positive N runs N samples and writes iteration-1/ … iteration-N/; when N > 1, the terminal summary covers only samples from the current command
--engineFrom configOverride engine name
--providerFrom configOverride engine.model.provider. When set, the complete --model value is preserved as an opaque model ID.
--modelFrom configOverride model. Without --provider, the legacy provider/name form is split when the prefix is configured; otherwise the complete value is preserved.
--parallelismFrom configOverride cases.parallelism. Allowed range: 1–256
--baselineFrom configOverride benchmark.enabled to true for this run
--api-key—Pass an API key (higher precedence than env vars)
--no-deletefalsePreserve workspaces or containers created by skill-up after evaluation for debugging.
--cleanup-sessionsfalseAlso delete the agent's session directory for the workspace after evaluation. claude_code and qodercli implement it today: they remove ~/.claude/projects/<workspace-key>/ and ~/.qoder/projects/<workspace-key>/ (or ~/.qoder-cn/projects/<workspace-key>/ for the CN edition), sessions and memory included. Off by default because it deletes history that today survives an evaluation; an explicit --cleanup-sessions=false overrides environment.cleanup_sessions: true, and --no-delete takes precedence over both.
--event-log—Write an ordered v1 JSONL evaluation event stream to a local file. The parent directory must already exist
--event-attribute—Attach a namespaced key=value string attribute to every event (repeatable; requires --event-log)
-v, --verbose0Increase log verbosity. Default info; -v / --verbose / --verbose=true → debug; -vv / --verbose=2 → trace; --verbose=false disables extra detail

Validation scope. run validates the eval-level config plus only the cases selected after --include-case-name / --exclude-case-name filters. Semantic validation errors in excluded cases do not block the run. File and metadata errors can still prevent filtering, as described below. Use skill-up validate to validate the whole suite (every case) regardless of filters.

YAML config loading rejects unknown fields, including nested struct fields and fields brought in by aliases or << merges. Errors name the field and line. Free-form maps such as engine.kwargs, environment variables and context.files still accept arbitrary keys. run reads case IDs before strictly decoding selected cases, using the filename when an ID is absent. Unknown fields and field-type errors in excluded cases do not block the run; unreadable files, malformed YAML, and invalid IDs still fail before filtering. A merged judge or judges block counts as present for explicit-null and mutual-exclusion checks.

Existing workspace state. --workspace deliberately reuses the same directory for every selected case, retry, and iteration. Runs are serialized, but configured setup/fixtures/skill installation and agent changes carry forward. The report --output-dir and --event-log must be outside this directory. Select one case and one iteration, with retries disabled, when each result must start from exactly the same state.

Examples ​

bash
# Run all cases
skill-up run ./evals/eval.yaml

# Run a subset
skill-up run ./evals/eval.yaml --include-case-name "basic-*"

# Evaluate an agent in an already prepared local workspace
skill-up run ./evals/eval.yaml --workspace /path/to/project --parallelism 1 --no-delete

# Exclude cases
skill-up run ./evals/eval.yaml --exclude-case-name "*-old" --exclude-case-name "*-deprecated"

# Override engine and model
skill-up run ./evals/eval.yaml --engine codex --model openai/gpt-4

# Preserve an opaque upstream model ID when its prefix is not a configured provider
skill-up run ./evals/eval.yaml --engine claude_code --model anthropic_modelscope/deepseek-v4-pro

# Preserve a slashed model ID unconditionally with a configured provider
export GATEWAY_API_KEY=your-key
export GATEWAY_BASE_URL=https://gateway.example.com/openai/v1
skill-up run ./evals/eval.yaml --engine codex --provider gateway --model team/model-v2

# Temporarily override case parallelism
skill-up run ./evals/eval.yaml --parallelism 4

# Run with baseline comparison
skill-up run ./evals/eval.yaml --baseline

# Multiple report formats
skill-up run ./evals/eval.yaml --format json --format html --format junit

# Three samples in this command, one folder per run, plus a simple flaky summary
skill-up run ./evals/eval.yaml --iteration 3

# Produce a live event stream that a CI process can tail while the run is active
skill-up run ./evals/eval.yaml \
  --event-log ./events.jsonl \
  --event-attribute com.example.ci.build_id=build-123

# Auto-detect mode (consumes Anthropic evals.json directly)
skill-up run --auto
skill-up run --auto --engine codex
skill-up run ./my-skill/ --auto

--event-log creates or truncates one file for the invocation. It cannot be used with --dry-run, and the file must not be placed inside a scheduled iteration-N/ output directory because that directory is recreated during the run. Consumers should read only newline-terminated records; the final record of a gracefully completed stream carries last_event: true.

Exit codes ​

Exit codeMeaning
0All cases passed
1At least one case failed/errored

Use the exit code in CI to determine whether the evaluation succeeded.

OTLP trace export ​

Telemetry is disabled by default. After standard OpenTelemetry env vars are set, skill-up run exports run traces over OTLP and decorates verbose slog logs with trace_id / span_id:

bash
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_METRICS_EXPORTER=otlp
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=local,service.namespace=skill-up
skill-up run ./evals/eval.yaml -v

You can also use trace-specific overrides like OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL. Currently grpc and http/protobuf are supported.

OTEL_METRICS_EXPORTER=otlp enables low-cardinality metrics (counters and durations for run/case/runtime exec). OTEL_METRICS_EXPORTER=console is also available for local debugging. OTEL_RESOURCE_ATTRIBUTES is forwarded as resource attributes. Set OTEL_METRICS_EXPORTER=none to explicitly disable metrics.


skill-up validate ​

Validate the eval config files. Run this before run to catch issues early. Unlike run (which validates only the selected cases), validate always checks the whole suite — the eval config and every referenced case file — ignoring any case-name filters.

It also loads every mocked MCP fixture declared by the eval or its cases and applies the same fixture checks as run. Missing files, invalid YAML and invalid tool definitions fail validation before any server starts. Real-server credentials and connectivity are not checked.

bash
skill-up validate [path to eval.yaml]

Examples ​

bash
skill-up validate
skill-up validate ./evals/eval.yaml

On success:

text
✓ eval.yaml is valid (loaded 3 case(s))

The validator checks that:

  • eval.yaml and every referenced case file exist and parse correctly
  • All required fields are present
  • Field values are within the allowed range

Skill content check (--skill) ​

bash
skill-up validate --skill <path to skill directory> [--uncited] [--strict]

Lints the content integrity of a single skill as authored on disk. It is a source-level check: it does not see what a run actually installs after skills.include/skills.exclude filtering, and it does not run automatically before evaluations. Findings print one warning per item:

  • SKILL.md opens with a closed YAML frontmatter block (--- fences)
  • the frontmatter declares non-empty name and description fields
  • every references/, assets/, scripts/ path cited in the markdown body exists on disk. The body is parsed as CommonMark: prose citations, local link/image destinations, and inline code spans that cite a path are checked, while code blocks, remote URLs, and raw HTML are ignored as documentation examples or external resources. Citations may be written with or without a leading ./, and equivalent spellings of one path (references/./guide.md, or references/my%20guide.md as a link destination) are normalized before comparison. File names keep all their characters: a link destination is resolved as CommonMark renders it (<references/my guide.md>, assets/diagram\(1\).png), and a local link is checked by its destination, not its link text; an inline code span is read literally (assets/icon@2x.png, or references/my guide.md when that file exists), less a trailing #anchor; prose citations may use letters of any script (references/指南.md). A space ends a path in prose, so cite a file name with spaces through a link destination or an inline code span. An inline code span that does not name an existing file is read as a command line, and only its first word, the script, is checked (scripts/build.sh --output assets/out.png checks scripts/build.sh alone). Arguments and further commands in the span are not citations, since they often name files the command creates; cite a file a command reads in its own code span or link

Pass --uncited to also report files under those directories that are not reachable from the body. Truncating the tail of a SKILL.md silently orphans the attachments its lost sections cited, and this report surfaces them. Reachability is transitive: a file a cited markdown document links to, or one a cited script mentions by a root-anchored or ./-relative path, counts as cited even when SKILL.md never names it directly. Dotfiles (.DS_Store, .gitkeep, dot directories) are local metadata and stay silent; a directory cited as a link destination (e.g. [docs](references/)) covers everything beneath it. The report is opt-in because reachability is a heuristic: a file a script loads through a computed path ($(dirname "$0")/lib.sh, a Python import), or one SKILL.md names only as a command argument, is reported even though the skill works.

A directory passed via --skill that has no SKILL.md is itself a finding. Pass --strict to promote warnings to failures (exit 1), including the --uncited report when both flags are set; without it the command exits 0 and only prints warnings. --skill cannot be combined with a positional eval.yaml path, and --uncited and --strict apply only to --skill.

On success:

text
✓ skill content check passed (./skills/my-skill)

skill-up list-cases ​

List every case referenced by an eval config — handy for quickly inspecting your suite.

This command strictly decodes every referenced case: unknown YAML fields and field-type errors fail loading. It does not perform semantic validation or run judges. Use validate to check configuration semantics.

bash
skill-up list-cases [path to eval.yaml]

Examples ​

bash
skill-up list-cases
skill-up list-cases ./evals/eval.yaml

Sample output:

text
ID                    Title                           Tag              Prompt
basic-success         Agent should find null bug      functional_test  Review the current diff and report ...
edge-case-empty       Handle empty input gracefully   functional_test  Review an empty repository with no ...
regression-001        Fix: no longer misreports       functional_test  Review the payment processor code ...

skill-up report ​

Regenerate reports from an existing result file without re-running the evaluation.

bash
skill-up report <path to result.json> [flags]

Flags ​

FlagDefaultDescription
--formatjsonReport format: json / junit / html / markdown (repeatable)
--output-dirSame dir as result.jsonOutput directory; created if missing

Examples ​

bash
# Generate an HTML report from existing results
skill-up report result.json --format html

# Multiple formats at once
skill-up report result.json --format json --format junit --format html --format markdown

# Generate a Markdown summary for CI or a PR comment
skill-up report result.json --format markdown

# Pin the output directory
skill-up report result.json --format html --output-dir ./reports

Report formats ​

FormatFileUse case
jsonreport.jsonMachine-readable structured data; consumable by downstream tools
junitreport.xmlJUnit XML; parseable by CI systems (Jenkins, GitHub Actions, …)
htmlreport.htmlHuman-readable visualization; open in a browser
markdownreport.mdConcise Markdown summary for CI logs and GitHub PR comments

For single-run reports, markdown is limited to skill-up report as an offline conversion format for an existing result.json. It is not currently accepted by skill-up run --format, skill-up debug report --format, skill-up debug judge --report, or eval.yaml report.formats.


skill-up compare ​

Compare two or more existing canonical result.json (or equivalent report.json) files without running evaluations or calling any Agent or Judge.

bash
skill-up compare <result.json> <result.json>... [flags]

The first file is the reference for deltas and gates. Every remaining file is compared independently against it, preserving argument order. This comparison role does not determine its display name. Any failed comparison makes the command fail; inputs are never compared as a sequential chain.

Flags ​

FlagDefaultDescription
--formattexttext, json, html, or markdown
--outputstdoutWrite the comparison to this file; its parent directory must exist
--baseline-labelautomaticOverride the first run's display label
--candidate-labelautomaticOverride later runs' display labels in input order; repeat once per later input or omit the flag
--fail-on-regressionfalseReturn a non-zero exit code for any shared case moving from PASS to another status
--max-token-increase-percentdisabledMaximum permitted increase in run-level total tokens (finite, non-negative percentage)

Examples ​

bash
# Compare several finished runs in one self-contained HTML report
skill-up compare baseline/result.json skill-a/result.json skill-b/result.json \
  --format html --output comparison.html

# Two-run comparisons use the same command and report structure
skill-up compare old/result.json new/result.json --format html --output pair.html

# Markdown tables for PR reviews and CI artifacts
skill-up compare old/result.json new/result.json --format markdown --output comparison.md \
  --baseline-label "Released Skill" --candidate-label "Proposed Skill"

# Explicit labels for multiple candidates (repeatable, commas remain literal)
skill-up compare baseline.json a.json b.json --format markdown --output variants.md \
  --candidate-label "Skill A" --candidate-label "Skill B"

# Produce a report even if the regression gate fails
skill-up compare old/result.json new/result.json --format html --output comparison.html \
  --fail-on-regression --max-token-increase-percent 20

# Stable machine-readable output
skill-up compare old/result.json new/result.json --format json

All formats use the same comparison and gates. Reports include run metrics, metadata, original source paths, a side-by-side overview of all runs, a case status matrix across the union of case IDs, and detailed fixed, regressed, changed, unchanged, added, and removed cases for each candidate against the baseline. The case matrix marks cases absent from a run explicitly; JSON uses null matrix cells. Matrix outcomes follow input order, and baseline-only cases are retained. JSON always uses the same top-level baseline, runs, case_matrix, comparisons, and gates structure, including for exactly two input files. comparisons contains one baseline comparison per candidate, with individual gate results; top-level gates aggregates failures and identifies the candidate. All runs have a one-based numeric ID (runs[].source.id) assigned in input order, including the reference. These IDs are also retained in each comparison's source identity; display labels do not change them.

Default display labels are the input paths exactly as supplied. Relative paths remain relative (including a leading ./ or ..); absolute paths remain absolute. Directory names are not shortened, and numeric IDs are not used as display labels. The source path is shown once, rather than repeated below the label.

Explicit label flags override path-based display labels. The first run is never automatically named Baseline, nor are later runs automatically named Candidate. The JSON keys baseline and candidate describe comparison roles only.

HTML extends the existing evaluation report with a Comparison tab and a run selector. In Comparison, the Reference and Compared selectors can choose any two inputs, in either direction. The selected pair shows metrics, metadata, coverage, and case transitions calculated with the same Go comparison logic. These controls do not change the first-input reference used by CLI/CI gates or text/JSON/Markdown. The case-response button opens the reused case viewer inside Comparison, showing both selected files' primary case results side by side: prompts, full responses (including recorded turns), grades and evidence. Case IDs absent on either side are explicitly marked absent. Cross-file comparison is read-only; normal Run details preserve each file's own with/without-Skill pairing and feedback. Pair summaries are pre-rendered for every ordered pair, so HTML size grows with the number of run pairs. No Agent or Judge is called when switching pairs. Selecting a run reuses the original Cases and Benchmark views, including prompts, responses, formal grades, and evidence. The existing single-run and with/without-Skill report presentation is unchanged. Comparison-mode feedback is isolated per run and exported locally, without using the single-run feedback API. HTML embeds its styles and filtering script, with no external assets or network requests; all cases remain readable without JavaScript. Markdown includes tables suitable for GitHub PR descriptions. Source result artifacts are preserved; output may not overwrite any input.

Cases align by case ID using the existing primary-result rule: prefer with_skill, then a non-baseline configuration, falling back to without_skill only when no other result for that case is available. In the usual with/without benchmark, cross-file comparison therefore compares each revision's with_skill result, not all variants against one another. This same selection rule drives metrics, the case matrix, transitions, and the interactive side-by-side viewer. Each file's internal with/without binding remains in Run details. The page keeps this rule in a collapsed explanation; the full selection semantics are documented here. Benchmark is the original single-run statistics view (including with/without deltas when recorded), and is shown in comparison mode only when the selected run actually contains benchmark data. The selected configuration is exposed in the matrix and transition tables.

For more than three inputs, overview/matrix columns retain readable widths and scroll horizontally with the first column pinned. Selectors wrap to new rows on narrow screens and reserve spacing for the dropdown arrow; the pair viewer always compares two selected inputs, regardless of the total input count. Benchmark reference entries do not double-count case counts or pass rates. Configuration identity is retained in case tables. Shared cases flag prompt changes when both prompts were recorded; otherwise the prompt comparison is unknown. Requested, applied, and observed agent configurations are compared when recorded. Total tokens are the recorded run-level total across all configurations; input/output tokens sum primary cases. Duration is run wall time, and pass-rate changes are percentage points.

Added and removed cases are shown separately, with shared-case coverage. Aggregate changes include each run's full primary case set, so different case sets are not paired samples. A change between two non-PASS statuses is classified as changed, not fixed or regressed. Recorded zero tokens remain zero; missing token metrics are unknown in human-readable reports and null in JSON, including deltas. The token gate fails when the total-token measurement is unknown, or when a zero baseline increases to a positive value.

Reports compare recorded outcomes only and do not regrade scores. Missing provenance is unknown; matching case IDs do not establish identical prompts, suites, Skill revisions, or judge/rubric definitions. Review those conditions before interpreting a change as an improvement. Gates are opt-in, apply equally to every output format, and write the report before returning a gate failure.


skill-up init ​

Write a skill-up user-config file. See User Configuration for the discovery chain and schema.

bash
skill-up init [flags]

Flags ​

FlagDefaultDescription
--localfalseWrite target is $PWD/.skill-up.yaml instead of the XDG path
--config—Source file to read (validated, copied verbatim with comments). Without this, a commented template is written.
--printfalsePrint to stdout instead of writing to disk
--forcefalseOverwrite an existing target

Note: for init, --config names the source to read. For every other subcommand (run, validate, …) it is the load-path override sitting at the top of the discovery chain.

Examples ​

bash
# Write a commented template
skill-up init                                  # -> ~/.config/skill-up/config.yaml
skill-up init --local                          # -> ./.skill-up.yaml
skill-up init --print                          # -> stdout

# Seed from an existing config (validates, preserves comments)
skill-up init --config ./team-config.yaml          # -> ~/.config/skill-up/config.yaml
skill-up init --config ./team-config.yaml --local  # -> ./.skill-up.yaml
skill-up init --config ./team-config.yaml --print  # validate & dump to stdout

# Overwrite
skill-up init --local --force

skill-up import ​

One-shot conversion of an Anthropic evals.json into skill-up's native YAML format.

bash
skill-up import <path to evals.json> [flags]

Flags ​

FlagDefaultDescription
--outputSame dir as evals.jsonOutput directory

Examples ​

bash
# Convert in place
skill-up import ./evals/evals.json

# Custom output directory
skill-up import ./evals/evals.json --output ./new-evals

The import produces:

  • eval.yaml — entrypoint config (with sensible defaults; review before running)
  • cases/*.yaml — one case file per evals.json entry

import vs --auto: import is a one-time format conversion — afterwards you maintain YAML files. run --auto consumes evals.json at runtime without producing intermediate files. See Migrating from Anthropic.


Output layout ​

After a run, the output directory looks like:

text
<skill-name>-workspace/
  iteration-1/                    # First iteration
    benchmark.json                # Aggregated stats
    <case-id>/
      with_skill/                 # Run with the Skill installed
        outputs/                  # Files generated by the Agent
          workspace/              # collect_artifacts matches (relative paths preserved)
        grading.json              # Grading result
      without_skill/              # Baseline (only when benchmark.enabled=true)
        outputs/
          workspace/
        grading.json

The outputs/workspace/ subtree appears only when collect_artifacts is configured; see Writing evals → Collecting workspace artifacts.

result.json configuration identity ​

The iteration-level result.json keeps engine_name and the requested-value model_name semantics for compatibility and also records credential-free objects:

  • requested_configuration: the engine, provider namespace, model, and version selected after YAML/CLI/credential precedence;
  • applied_configuration: the adapter protocol and model skill-up forwarded to the CLI. An empty applied model means model selection was delegated to local/default settings. Its provider is also empty when the adapter did not actively select one;
  • observed_configuration: present only when every runner session explicitly reported the same model. If absent, the runtime model is unknown. Per-case observed_model retains an individual agent-reported value when available.

Applied configuration is not proof of the CLI's final choice: local settings may override command-line or environment values. skill-up does not make an extra model request merely to inspect local login state. When a Codex custom provider cannot be applied, its provider-scoped endpoint and credential are omitted from the fallback invocation rather than being sent to Codex's local/default provider.

Capability warnings and requested/applied values are also attached to each agent SessionResult; its legacy model field is reserved for an explicitly agent-reported observation. API keys and other credential values are never written to these fields.

grading.json ​

Per-case grading result (Anthropic-compatible):

json
{
  "expectations": [
    {
      "text": "Output contains the keyword `null`",
      "passed": true,
      "evidence": "final_message contains 'null pointer at line 42'"
    }
  ],
  "summary": {
    "passed": 1,
    "failed": 0,
    "total": 1,
    "pass_rate": 1.0
  }
}

Note: the grading.json under the workspace uses the Anthropic-compatible shape — only expectations and summary at the top level. The full evaluation status (status, turns_executed, turns_total, assertion_results) lives under case_results[].grading inside result.json.

The grading object inside result.json carries the full status:

  • PASS — all assertions passed
  • FAIL — at least one assertion failed
  • ERROR — execution exception (timeout, engine crash, etc.)

benchmark.json ​

Aggregated statistics across all cases:

json
{
  "run_summary": {
    "with_skill": {
      "pass_rate": { "mean": 0.83 },
      "time_seconds": { "mean": 45.0 },
      "tokens": { "mean": 3800 }
    },
    "without_skill": null,
    "delta": null
  }
}

Released under the Apache 2.0 License.