Code & hybrid judges

A deterministic eval is a plain function#

1Reply with exactly APPLE.

1import type { JudgeContext } from "@hona/openeval";
2
3export default ({ response }: JudgeContext) => ({
4  scores: { correct_answer: response.text === "APPLE" },
5});

1import type { Benchmark } from "@hona/openeval";
2
3export default {
4  models: ["provider/candidate-model"],
5  repetitions: 3,
6} satisfies Benchmark;

The runner discovers judge.ts by convention. It receives JudgeContext and returns JSON, synchronously or asynchronously. response.text is always a string; missing text becomes an empty string. The execution outcome remains available on context.run.

The author chooses criterion IDs and all grading rules. No task-specific scorers or builder APIs are required. Custom JSON is retained as-is. An output without scores is unscored and cannot silently disappear from the benchmark denominator.

Both files contribute to one eval#

1Summarize the incident in ./incident.txt. Return a JSON object with a "summary" string.

1## Criterion: supported_summary — Source-supported summary
2
3Pass when the summary covers the incident's impact and resolution
4without contradicting the supplied report. Fail for missing or false
5information. Accept equivalent concise wording.

1import type { JudgeContext } from "@hona/openeval";
2
3export default ({ response }: JudgeContext) => {
4  try {
5    const value = JSON.parse(response.text);
6    return {
7      scores: { json_shape: typeof value?.summary === "string" },
8    };
9  } catch {
10    return { scores: { json_shape: false } };
11  }
12};

Supply incident.txt in the workspace. The LLM grades supported_summary, while code grades json_shape. Both read the same candidate recording. Duplicate criterion IDs are errors; neither source overwrites the other. A hybrid judgment becomes active only after both sources succeed.

All the recorded data, with lazy native access#

1import { readRecording } from "@hona/openeval";
2
3await using context = await readRecording("./results/RUN", "eval_ID");
4const messages = await context.recording.messages();
5const db = await context.native.database();
6console.log(db.query("SELECT name FROM sqlite_master").all());

The runner owns reader lifetimes and initializes native services only when requested. Native reads are bound to the recorded database, not your live OpenCode service. The SDK opens supported native schemas on disposable copies; recorded and reader versions remain distinct metadata.

1const result = await context.verification.run({
2  revision: "final", timeoutMs: 60_000,
3  commands: [["bun", "/verification/check.mjs"]],
4  files: { "check.mjs": checkSource },
5  artifacts: ["checks.json", "screenshot.png"],
6});
7
8return { scores: { correct: {
9  value: result.state === "completed" && result.exitCode === 0,
10  reason: "Explain the verified outcome.",
11  evidence: [{ kind: "verification", id: result.id }],
12} } };

Set judge.verification.image in benchmark.ts, then build the standard image with openeval image --verification. The image includes Bun, Python, and Playwright with Chromium. Check inputs remain outside the candidate. Artifact tests and thresholds belong to your judge, not the SDK. Boolean and numeric scores still work; structured scores optionally add reasons, evidence, and measurements.

Common measurements are automatic#

A successful shell tool returning a failed test is distinct from a native tool failure. Counts describe recorded native invocations, not inferred operations inside a batch. Missing usage is unavailable. Reported zero cost does not establish free service.

Token categories keep their native meanings. Avoid double-counting reasoning/output or cache categories. These observations affect a score only when the author's function or rubric explicitly uses them. Markdown judges can inspect the same primitives through candidate_evidence with action metrics.

Frozen inputs, bounded execution, inspectable results#

The host bundles local imports without executing the judge during planning. Its fingerprint covers the bundled code and the versions of imported packages, not the checkout location or lockfiles. Code and reference changes schedule rejudging; the candidate recording is reused.

judge.ts runs in a separate Bun process under judge.timeoutMs, so the host can stop synchronous loops. Code and hybrid evals grade finalized recordings; earlyStop is supported for Markdown-only evals. The viewer shows returned JSON, frozen source, process logs, and candidate metrics.

1import { recordEvidence, judgeEvidence } from "@hona/openeval";
2
3const evidence = await recordEvidence({
4  directory: "./controls/apple/evidence",
5  prompt: "Reply with exactly APPLE.",
6  response: "APPLE",
7});
8const result = await judgeEvidence({
9  evidence,
10  code: "./evals/exact-answer/judge.ts",
11  directory: "./controls/apple/judge",
12});

Code-only controls use no model calls unless the author writes one explicitly. The original JSON is retained, while normalized scores are stored with their source. Failed grading does not overwrite finalized judgments.