# Run your benchmark

Plan a small batch, control cost, and keep one aggregate.

<a id="small-batch"></a>

## Start with a scoped plan

### terminal

```shell
bunx --bun @hona/openeval plan \
  --only-eval ask-dialect \
  --only-model provider/candidate-model \
  --only-repetition 1

bunx --bun @hona/openeval run \
  --only-eval ask-dialect \
  --only-repetition 1
```

| Scope | What stays in the result |
| --- | --- |
| --only-eval | Other evals and their active scores |
| --only-model | Other models and their active scores |
| --only-repetition | The configured repetition count |
| No scope flags | All missing or changed work |

Repeat a scope flag to select multiple values. Scope controls execution, not the benchmark's denominator.

<a id="resume"></a>

## Reuse evidence deliberately

| What changed? | Next run |
| --- | --- |
| Nothing | Reuse the selected candidate and judgment |
| Prompt, workspace, or candidate runtime | Collect new candidate evidence for affected inputs |
| Rubric or shared judge instructions | Rejudge the retained recording |
| A new model | Add its eval / repetition slots |
| The runner stopped mid-run | Close its open sessions as interrupted and collect them again |
| --new | Create a separate benchmark result |

> **One runner per aggregate**
>
> Let an active invocation finish before extending the same result with another invocation. If a runner stops without finishing, run, retry, and rejudge wait until its heartbeat is two minutes old, then recover. Other failed executions require an explicit retry or rejudge.

<a id="budget"></a>

## Bound candidate and judge work

### terminal

```shell
bunx --bun @hona/openeval run --max-cost 5
bunx --bun @hona/openeval run --final-only
```

| Control | Meaning |
| --- | --- |
| concurrency | Shared candidate / judge worker limit; default 10 |
| --max-cost | Admission budget using reported spend and reservations |
| candidate.timeoutMs | Agent session limit: 45 minutes by default, at most 12 hours. Set it in benchmark.ts, or in eval.ts for one eval |
| candidate.maxCostUSD | Stops an attempt once its recorded model cost reaches the limit; the run is recorded as cost limited and judged like a timeout. One in-flight step can overshoot |
| --final-only | Judge after the candidate finishes |

A scheduling budget stops admitting new work. Active sessions finish normally, so final billing can exceed the admission budget. Work without a cost estimate can be deferred.

### eval.ts · opt-in early stopping

```typescript
import type { Eval } from "@hona/openeval";

export default {
  earlyStop: { minIntervalMs: 45_000, maxChecks: 12, maxCostUSD: 1 },
} satisfies Eval;
```

Early stopping requires every criterion score to be irreversible and non-null. Monitoring cannot steer the candidate. If monitoring pauses or exhausts its budget, final grading still follows execution. If you turn early stopping off, the next run collects new sessions only where it stopped one. Sessions that finished normally stay reusable.

[Documentation index](https://openev.al/llms.txt)