Your first eval

1. Check the tools#

1bun --version
2docker info
3opencode --version

Install BunInstall DockerInstall OpenCode V2

If your V2 executable is named opencode2, use that name for the OpenCode commands below.

2. Pick a directory#

Use an existing benchmark package or an empty project. For a new one:

1mkdir my-benchmark
2cd my-benchmark
3bun init -y
4bun add --exact @hona/openeval@0.5.3
5bunx --bun @hona/openeval --version

Keep the following commands in this directory. In an existing package, keep its manifest and add the dependency with its package manager.

1node_modules/
2results/

3. Add the writing skill#

Download eval-writing.zip

Extract it into your project root. Keep an existing customized skill if you already have one.

1.opencode/skills/eval-writing/
2  SKILL.md
3  README.md
4  references/
5    openeval.md
6    examples.md
7    research.md

Open this project in OpenCode. Use /eval-writing when you want help with your own task or rubric.

1opencode .

4. Write one eval#

An eval is a task plus its grading rules. The candidate does the task; the judge grades its recording. Each criterion is one scored requirement.

Try this SQL demo: does the candidate ask which database you use, and handle the customer ID safely? Create these files:

1benchmark.ts
2evals/
3  ask-dialect/
4    prompt.md    # the task
5    judge.md     # the criteria

1Write a SQL query for the ten most recent orders for a customer.

1# Requests the SQL dialect
2
3## Criterion: asked_dialect — Asks for the SQL dialect
4
5Pass when the agent asks which database or SQL dialect is in use.
6Fail when it assumes a dialect without asking.
7Asking alongside a draft counts.
8
9## Criterion: safe_parameters — Uses bound parameters
10
11Pass when the proposed query uses a bound customer-ID parameter
12and explains how to supply its value.
13Fail when it interpolates customer input into SQL
14or does not provide a parameterized query.

Task promptsRubricsCode judgesWorkspace files

5. Choose models and variants#

In OpenCode, connect your provider and pick a candidate and a judge:

1/connect
2/models

Replace both placeholders below with connected model IDs. A benchmark groups your evals and model choices.

1import type { Benchmark } from "@hona/openeval";
2
3export default {
4  models: ["provider/candidate-model"],
5  candidate: { websearch: false },
6  judge: { model: "provider/judge-model", websearch: false },
7  repetitions: 1,
8  concurrency: 1,
9} satisfies Benchmark;

Use provider/model#variant for a listed variant, such as #high. Omit the suffix for the default. Start with one candidate and one repetition. This demo needs no web search.

Using only judge.ts files? Omit judge.model; you only need the candidate model.

Models & variantsBenchmark type

6. Plan, then run#

1bunx --bun @hona/openeval image
2bunx --bun @hona/openeval plan --only-eval ask-dialect --only-repetition 1

Check the plan: ask-dialect, your candidate model, repetition 1. Planning previews work and its cost estimate. When you are ready to call the models:

1bunx --bun @hona/openeval run --only-eval ask-dialect --only-repetition 1

Running later? Keep the files and use the same command when ready.

Run scopes & budgetsCLI reference

7. Watch and inspect#

To watch live, open a second terminal in the same directory. You can also open the viewer after the run.

1bunx --bun @hona/openeval view

Open the results viewer

1. Select ask-dialect
2. Select your model
3. Read Session / Judge
4. Follow the evidence

Next: add your other model IDs or increase repetitions in benchmark.ts, then plan and run again. OpenEval resumes the same result and keeps completed work.

Read the evidenceScoringDefinitions