How OptBench decides: switch, keep today's model, or test more

This page describes what the code does today. Where a rule has a number, the number is the one the code uses.

Words used on this page

• Use case: one AI call site in your code, with its prompt, the model it uses today, its metrics and its test cases. A use case is a single call, an agent loop with tools, or a pipeline of several calls.
• Today's model: the model the use case uses in production now. It answers every case in the same run as the candidates, as a reference, and is never the one recommended.
• Candidate: a model being tested. You pick candidates, open or closed. If you pick none, OptBench proposes models that can do the task and that your model rules allow.
• Case: one test input, for example a user message, a filled template, or a task for an agent.
• Reference: what a good answer looks like for a case: an exact answer, a label, a JSON object, or checks such as "must include" or "required tool calls".
• Gold: a reference that a person wrote or approved. Only gold counts toward a Decision.
• Gate: a hard limit on a metric, for example an error rate of at most 2%. A model that fails a gate is not ranked.

1. One use case per AI call

Your coding agent connects to OptBench over MCP and scans the repo. Each place the code calls a model becomes its own use case, because a model that is good enough for a classifier can be wrong for an agent that issues refunds. The agent proposes metrics for each use case from the metrics catalogue, for example json_schema_valid for structured output or agent_task_success for a tool loop, and stops at set points to ask you before anything is saved or run.

2. Test cases and their references

A case can come from five places:

• your coding agent, reading the repo and its tests
• a person, in the dashboard
• OptBench's case generator
• a file you upload (CSV, JSONL, images)
• your production traces (PostHog, Langfuse, OpenTelemetry or a trace file), redacted before anything is stored

The answer your production model gave in a trace is kept for comparison, but it never counts as the reference. A reference written by your coding agent or drafted by a model scores answers, but it is marked as written, not approved, and does not count toward a Decision. It becomes gold when a person approves it, in the dashboard or by telling their agent to. Editing a case's input or reference removes its approval.

3. The run

• An estimate is free and makes no model calls. A run starts only when someone confirms the estimated cost, and it stops at 1.2 times the confirmed amount.
• Every candidate and today's model answer the same frozen set of cases. For a case imported from production, today's recorded answer is reused for the first repeat, at no cost.
• Agent loops run against recorded or mocked tool results, so a candidate never acts on your real systems.
• Repeats are optional. With two or more, OptBench also reports how often a model passes every time and how often its result flips on the same case.

4. Scoring and judging

Two kinds of metrics score each answer:

• Deterministic metrics run in code: exact match, JSON schema validity, required and forbidden tool calls, "must include" checks, and so on. Latency, cost, and error, throttle and truncation rates are measured on every run.
• Judge metrics use a model to grade: a rubric score, and a pairwise comparison with today's answer.

The default judge is Jev 1.14, a decision model: it answers typed questions instead of chatting. When Jev can't judge an answer (a system prompt or task over 8,000 characters, an image it can't read, an error or an abstention), a chat model judges it instead. A juror never votes on an answer from its own model family. A rubric score is the mean of the votes. A pairwise vote is 1 if every juror prefers the candidate, 0 if every juror prefers today's answer, and 0.5 otherwise. When jurors agree on less than 70% of the answers they judged, the run is flagged with low judge agreement. The research article on one judge vs a jury covers why OptBench uses one decision model by default.

Each answer gets one quality score, the weighted mean of the quality metrics it has (rubric 0.6 and pairwise 0.4 when both apply). An answer passes when the call didn't error, its quality is at least 0.8 (the default success threshold) and every per-case gate holds. For agent loops, passing means agent_task_success is 1. Cost per successful task is everything the model cost in the run divided by the number of answers that passed.

5. Gates and the leaderboard

Gates are hard limits on a metric. A model that fails one is not ranked. If today's model fails the same gate, the gate becomes a soft gate: it is reported but doesn't block, since it would block every model. The leaderboard orders the models that pass every gate by a utility that weighs quality, speed and cost (60/20/20 by default). The verdict below decides which model is named.

6. The verdict

The floors

A verdict can be a Decision only when both hold:

• N ≥ 30: at least 30 distinct cases were compared (candidates and today's model, errors included).
• k ≥ 50% of N: at least half of those cases have gold.

Below either floor the verdict is a Shortlist, and its reasons say how many more cases or approvals are needed: max(0, ceil(0.5 × N) − k) approvals.

When today's model is measured

This is the usual case: the AI call is in production and today's model answered in the run.

• For each case, take each model's quality, averaged over repeats. An errored call counts as 0.
• For each candidate, take the difference with today's model on every shared case, and run a one-sided paired bootstrap with 2,000 resamples. The p-value is the share of resampled mean differences at or below zero.
• Correct for testing several candidates at once with Holm's step-down method at α = 0.025, one-sided, over all candidates.
• A candidate beats today's model when it clears that test and passes every gate.

If the floors are met and at least one candidate beats today's model, the verdict is Switch to A, where A is the one with the lowest cost per successful task among them. If none does, the verdict is Keep today's model. If the floors are not met, the Shortlist lists every candidate that beat today's model, or, when none did, up to three candidates that pass the gates, ordered by quality, leaving out any that is both costlier per successful task and slower than today's model.

Before launch, with no model in production

There is no today's model to compare with, so the test is a quality bar instead:

• A candidate clears the bar when it passes every gate and the lower end of its 95% quality interval is at least 0.8.
• If the floors are met, the verdict is Use A, where A is the cheapest per successful task among the models that clear the bar.
• If not, the Shortlist lists those models, cheapest first. If none clears the bar, the verdict says nothing is worth testing yet.

Every verdict stores how many cases were compared, how many had gold, its reasons, and the cost per successful task of the named model with a 95% interval over cases.

Statistics at a glance

After you approve references

Approving references doesn't re-run anything. A rescore is free: it recomputes the deterministic metrics, gates and verdict with the current references and approvals, and reuses the judge scores already stored. A run whose prompt or cases changed needs a new run instead, and OptBench says which.

What a verdict does not claim

• It is about your cases, not every input. Each verdict says what it was measured on, for example "measured on 42 cases, 24 with a reference". More varied cases make it more useful.
• A Shortlist is not a recommendation to switch. It names the models worth testing next and what is missing for a Decision.
• Keep today's model means no candidate showed a quality gain on these cases after correcting for the number of candidates. It does not show that the candidates are worse, or that no better model exists.
• Before launch, the quality bar is the only quality test. Models that clear it are not compared with each other for significance; the cheapest one is named.
• Judge scores are a model's judgment. References and deterministic checks make them more reliable, and low agreement between jurors is flagged on the run.
• Cost and latency are what the run measured on the hosts it used.