Write judge rubrics

Declare each criterion#

1# Requests the SQL dialect
2
3## Criterion: asked_dialect — Asks for the SQL dialect
4
5Pass when the agent asks which database or SQL dialect is in use.
6Fail when it assumes a dialect without asking.
7Asking alongside a draft counts.
8
9## Criterion: safe_parameters — Uses bound parameters
10
11Pass when the proposed query uses a bound customer-ID parameter
12and explains how to supply its value.
13Fail when it interpolates customer input into SQL
14or does not provide a parameterized query.

Group criteria into categories#

1## Criterion: asked_dialect — Asks for the SQL dialect
2Categories: misalignment

1export const criteria = {
2  correct_answer: { name: "Correct answer", categories: ["general"] },
3} satisfies CodeCriteria;

A category is any string. Put the optional Categories: line directly below a heading, or export criteria from judge.ts. OpenEval removes category lines before the judge reads the rubric, so adding or changing categories never rejudges recorded evidence.

The viewer adds a Categories tab with a radar chart and a models-by-categories heatmap, and a filter that rescores any view to the selected categories. Benchmark results also show a scorecard: the benchmark on the left, one row per category, and one column per model. Export image saves that table or the totals chart as a PNG. Set categories in benchmark.ts to compose a benchmark from them, or pass --only-category to run a subset.

Decide what counts#

Let the shared judge handle mechanics#

1. Read the rubric
2. Inspect the recording
3. Cite evidence
4. Submit every criterion score

The native openeval-judge agent supplies the common evidence, citation, and submission instructions. Your rubric supplies task-specific criteria. Structured tools validate the judge's submission and return errors for correction.