Independent multi-judge evaluation

Status: sequential shared-runtime execution. The evaluated Agent runs once; judges reuse its runtime with fresh Agent sessions and separate verdicts/artifacts. Numerical scores, configurable policies and validator bundles remain future work.

Related issue: #246. Usage: Writing evals.

Problem and goals

A generated change may need deterministic validation and semantic review. Running the evaluated agent twice changes the input being evaluated. Asking an Agent judge to run a script also leaves script execution and interpretation under that agent's control: it does not inherit the framework's script exit-code verdict, timeout handling, or independent attribution.

Run the evaluated agent once, then preserve an independent verdict for each named judge. A script failure must remain visible even if semantic review passes. Extra Agent criteria must not silently increase that judge's weight in the case verdict.

Configuration and compatibility

The snippet uses the OpenCode/DashScope engine and endpoint configuration from the code-stats example. An Agent judge inherits that engine and endpoint and requires an explicit model.

judges:
  - id: functional
    type: script
    script_path: evals/scripts/check.sh
  - id: semantic
    type: agent_judge
    model: dashscope/qwen3.8-max
    criteria:
      - The change addresses the request without unrelated modifications.
    pass_threshold: 1.0

Each member has a unique ID. Eval-level lists supply defaults; a case-level list replaces the entire list. A case-level singular judge also replaces an inherited list. A document cannot contain both judge and judges. Legacy singular configs retain their execution and reporting behavior. expect is a separate hard gate. No public aggregation-policy configuration is introduced by this implementation.

Execution and shared state

1. Resolve and validate the judge plan.
2. Execute the evaluated Agent and run expect gates. A failed gate skips all members and produces a failed case with gate evidence.
3. Run each member sequentially in the same case runtime. Each Agent judge starts a fresh session using the original transcript and execution metadata rather than resuming the evaluated Agent or a sibling judge.
4. Preserve each member's outcome and artifacts before the next member starts. A completed member's FAIL or ERROR does not prevent later members from running. Case cancellation skips pending members; member deadlines stay bounded by the case deadline.
5. Aggregate the case and clean up its runtime according to the configured policy.

This follows Harbor's shared verifier environment model. Dependencies, volumes, background services, environment variables and network policy remain available. No snapshot, sandbox restore or filesystem-copy capability is required. The filesystem is shared: a judge's writes, installed Skills and external side effects can affect later members. Use read-only checks when judges must see unchanged output; separate verdicts and sessions do not imply filesystem isolation.

List order is execution order. One member's verdict or conversation is not passed to another member; filesystem changes remain visible. The original Agent is not rerun. Context and session artifacts are archived under each judge's ID before later calls can overwrite runtime-backed files.

Verdict and result contracts

judge.OutcomeAggregator separates policy from execution. Its Strategy names the policy and Aggregate combines leaf outcomes into compatibility grading. DefaultOutcomeAggregator selects AllRequiredAggregator; the evaluator records that strategy and the report uses it rather than inventing its own policy name. A gate failure retains its own case status; skipped member slots still determine the compatibility scoring denominator.

The current policy is all_required:

• PASS: every executed member passes.
• FAIL: at least one member fails and no member has ERROR or SKIP.
• ERROR: at least one member errors or is skipped after judging begins.

Case pass rates count PASS verdicts across cases. Anthropic-compatible benchmark pass rates average per-run compatibility assertion pass rates; these metrics are not interchangeable. In the legacy single-judge path, the configured denominator includes expect rule types plus judge assertions. A failed expect gate preserves skipped judge assertions with no credit and skipped: true, rather than shrinking the denominator or inventing a judge failure. Agent-judge thresholds still use judge criteria alone.

Each member contributes one assertion to compatibility grading, including when an expect gate skips all members. Skipped members receive no credit and remain SKIP in judge_results; the failed gate sets the overall case to FAIL. Expect checks and leaf criteria do not add weight to multi-judge compatibility grading. Leaf criteria remain in that member's result, including permitted failed criteria when an Agent judge meets its pass_threshold. Script exit 0 means PASS; nonzero means FAIL; execution failures and strict multi-judge timeouts mean ERROR.

evaluation.json version 1 is the canonical grouped artifact: gates, judge_results, and aggregation (strategy and status). Members retain ID, type, status, result/evidence, errors, skip reason, duration, and artifact references. grading.json is a compatibility projection; its pass rate is the fraction of passing member verdicts, not an arbitrary numerical quality score. Reports retain the group structure. Existing benchmark summaries remain based on case verdicts.

Comparison with Harbor

Reference: Harbor main at 58789aad577f432a4590209a8fd901f8c894f382. This comparison is based on source inspection, not runtime experiments.

Layer	Harbor	skill-up in this PR
Evaluation output	Named numerical rewards	Named judge verdicts and evidence
Default script protocol	reward.json or reward.txt	Exit code and diagnostic output
Attribution	Reward map itself contains numerical values	Each judge has execution status and artifacts
Aggregation	Metrics across trial rewards; separate multi-step policy	all_required within one case
Custom implementation	Imported verifier class	Built-in judge factory; scripts as external validators
Environment	Abstract environment; shared or separate verifier	One shared case runtime; fresh Agent judge sessions
Supporting files	Tests directory upload or image-owned tests	Script entry-path reuse; no validator bundle contract

Harbor's default verifier reads numerical rewards after script execution. Multiple reward keys do not by themselves represent independently executed judges with individual deadlines and errors. Its verifier factory also accepts custom verifier classes.

Harbor's metric aggregation aggregates across trial reward maps, treating missing maps or keys as zero. Its multi-step aggregation supports per-key mean or final-step reward; steps without verifier results are excluded from the mean denominator. These policies are distinct from independent judges over one agent execution. A separate verifier environment receives collected artifacts rather than an automatic copy of the complete agent workspace.

Future extension boundaries

The following are proposals, not supported YAML or result fields:

1. Numerical judge scores. Add optional named finite scores alongside verdicts and evidence, with explicit scale and direction. Keep execution errors distinct from legitimate zero scores and keep assertion pass rates unchanged. Specify result-version compatibility before extending serialized artifacts.
2. Case policies. Select an OutcomeAggregator through a validated configuration contract when concrete use cases require weights, thresholds, or required versus optional judges. Define ERROR, SKIP, missing-score and denominator semantics; never infer weights from the number of leaf criteria. Update report and benchmark consumers with the policy, not just the evaluator.
3. Benchmark metrics. Aggregate numerical dimensions across cases and repeated runs separately from case acceptance. Define missing-value and failure policies explicitly rather than copying Harbor's zero substitution implicitly.
4. Declared inputs. Artifact-only verification can be a separate input contract. Durable regrading additionally needs retained inputs, configuration and validator identities; shared runtime execution is not durable replay.
5. Reusable validator bundles. Extend script-path reuse with supporting modules, data, explicit parameters, dependency/environment requirements, and versioned JSON input/output. Retain exit-code compatibility. Directory packaging and installation/readback must be verified together.
6. External judge adapters. Prefer a versioned process protocol when an external implementation is needed; adding a Go interface does not create a public plugin ABI. The current judge and runtime types remain internal.

Validation boundaries

Tests cover shared filesystem state, fresh judge sessions without cross-member resumption, one evaluated-Agent execution, gate short-circuiting, cancellation, timeout/error handling, grouped reports and compatibility grading. Runtime reuse has no snapshot capability gate. The gated Docker integration test covers two Agent judges plus a script, outside-workspace dependencies, runtime identity and material readback, including script FAIL / Agent judges PASS / aggregate FAIL. Gated OpenSandbox tests cover shared-runtime fixtures, real OpenCode provider requests, and configuration-only default-provider readback. These tests require explicit flags and service credentials; skipped cases do not prove live behavior.

The historical validation below predates shared-runtime execution and records what was verified for PR #290; it does not describe current filesystem isolation.

Live Qwen verification

On 2026-09-30, the existing code-stats example was run through OpenCode 1.14.24 with dashscope/qwen3.8-max, using DashScope's OpenAI-compatible endpoint. These were real model calls, in addition to the mocked-agent regression tests above. The model identity is the requested and applied client configuration; the provider's serving implementation was not independently audited.

The selected analyze-directory case reuses the existing check-stats.sh without modifying it and adds an independent semantic judge. The fixture contains three specified files, totaling 20 lines and 350 bytes. Tool configuration and installed Skill files are excluded from that scope.

Run	Expect gates	Format script	Semantic Agent	Aggregate
With Skill	PASS	PASS	PASS	PASS
Without Skill baseline	FAIL	SKIP	SKIP	FAIL
Controlled script failure, with Skill	PASS	FAIL	PASS	FAIL

The positive run reported the expected file, line and byte counts. The baseline failed the exact Total Files: 3 and Total Lines: 20 strings; this is a formatting gate result, not evidence that the model cannot calculate those values. Gate failure correctly skipped both judges.

For the controlled negative run, a temporary copy of the example disabled the baseline and changed only the script's final success path to write a .judge-isolation-probe file and exit 1. An extra semantic criterion required the probe to be absent. The Agent judge's recorded directory checks found no probe in its workspace; it continued and passed, while the script retained FAIL and all_required returned FAIL. This verifies workspace mutation isolation and failure attribution with a real Agent judge. It does not claim host or network isolation.

Both runs preserved evaluation.json, compatibility grading.json, agent and judge transcripts, and the semantic context manifest. Included context paths were resolved against the archived outputs and read back successfully. JSON, HTML and JUnit reports were generated for the positive run.

To repeat the positive run, inject DASHSCOPE_API_KEY securely, then run from the repository root:

make build
./bin/skill-up validate examples/code-stats/evals/eval.yaml
./bin/skill-up run examples/code-stats/evals/eval.yaml \
  --include-case-name analyze-directory --iteration 1 --parallelism 1 \
  --output-dir /tmp/code-stats-qwen-positive \
  --event-log /tmp/code-stats-qwen-positive-events.jsonl \
  --format json --format html --format junit

To reproduce the negative test, copy examples/code-stats to a scratch directory, set benchmark.enabled: false, replace the final exit 0 in the copied check-stats.sh with the following, and add the probe-absence criterion to the copied case's semantic judge:

printf 'script-only mutation' > .judge-isolation-probe
echo 'CONTROLLED_FAILURE: functional judge deliberately exits 1'
exit 1

Run the copied eval with the same case selector and a separate output directory. Expect process exit 1, script FAIL, semantic PASS and aggregate FAIL. Inspect the semantic transcript for the actual directory check rather than relying only on its verdict. Keep this injection out of the normal example.

Both scenarios were rerun after fixing snapshot modification-time preservation, with the same verdicts and successful archived-material readback. A regression test additionally preserves source/output freshness and directory modification times across capture and independent forks.

Only this selected case and the controlled negative variant were run against the real model. The other three code-stats cases and the six-case skill-upper suite were configuration-validated but were not run against Qwen in this verification. The Harbor comparison remains a source comparison, not a live Harbor experiment.