Benchmark before launch

Pick the model before your users see it. Start here: you need no production traffic, only your code and a few cases you trust. Once you are live, the same use cases and cases carry over to Connect traces and Monitor in production.

The steps: scan the repo, one use case per call site, 2 or 3 cases by hand, generated cases up to about 30, a pilot, an estimate, the run.

Scan the repo

Open your product repo and ask your agent:

Scan this repo with OptBench.

The agent calls biy_scan_repo and follows the instructions it returns. It finds every LLM call, agent loop and pipeline, and creates one use case per call site (biy_use_cases_create, then biy_use_cases_fill for each) with the real prompt, tools, output schema, metrics and gates. Prompts are copied exactly, never shortened or paraphrased. The scan also proposes model rules from what the code shows, such as the providers and regions you call (biy_propose_model_rules). You approve them in the dashboard.

One use case per call site

Two calls with different prompts get two use cases, even when they use the same model today. A model can be right for one and wrong for the other, and each gets its own verdict. A pipeline with model slots (a small and a large model, say) is one use case with a slot per model.

Write 2 or 3 cases by hand

The cases you write are the ones you trust most. Take real inputs: a typical one, a hard one, and one that should be refused or fail a check. Write the right answer for each. In the dashboard: the use case's Cases tab. With your agent: biy_cases_add.

A case you write or upload with its answer counts as "with a reference", which a Decision needs (below).

Generate the rest, up to about 30

biy_cases_generate writes more cases seeded from the ones you have, in the background (1–3 minutes). Aim for about 30 cases per use case. The more cases, the more accurate the result: on 10 cases, two models a few points apart look the same.

Generated cases have no answer a person wrote. They count as cases, not as "with a reference". To raise that share, write gold for some of them: Write gold.

Check, pilot, estimate

• biy_use_cases_check lists what a use case still needs (a prompt, tools, cases, gates). Fix that before anything costs money.
• A pilot runs 2 cases once on every model. It costs a few cents and measures what each model really writes, so the full estimate afterwards uses measurements, not token guesses. Use it for a use case that has never run, or one that writes long answers. Estimate it with biy_runs_estimate and pilot=true; it gives no verdict.
• biy_runs_estimate is free: the models, the number of calls and the cost of the full run. For several use cases at once: biy_runs_estimate_many.

The run benchmarks your workspace's candidate models. When the use case already has a model (a closed one, say), that model runs on the same cases as the baseline: the verdict compares every candidate with today.

Run

Your agent shows you the estimate and asks. Only after your yes does it call biy_runs_start with confirm_cost_usd set to the amount you agreed to. A run never spends more than that, and never more than the workspace's per-run cap. biy_runs_status shows progress, the leaderboard so far and the verdict when the run is done. To add models later without starting over: biy_runs_extend.

Shortlist or Decision

Every verdict has a tier.

"The models separate" means a paired bootstrap over the cases (95% interval):

• Your call is live (today's model measured): a candidate beats today when its quality gain per case is above zero across the whole interval and it passes the gates. The cheapest of those, by cost per success, gives "Switch to A". None gives "Keep today", and so does a tie.
• Before launch (no today's model): a model survives when the low end of its quality interval reaches your success threshold and it passes the gates. One survivor gives "Use A". Several give "Use A" only when the best one separates from every other; otherwise "Worth testing" lists them all.

What counts as a reference: a case you wrote or uploaded with its answer, or gold a person approved. Generated cases, references from the scan, gold that no person has approved yet and today's production answers do not count.

Every verdict, on both tiers, shows its evidence:

measured on N cases, k with a reference

It also shows the cost per successful task of the verdict's model, with its 95% interval, and the tier reasons: what would turn a Shortlist into a Decision (more cases, more references, a clearer gap). A Shortlist makes no monthly saving claim.

After the run

• Shortlist: add cases (biy_cases_add, biy_cases_generate), write gold (Write gold), then grow the run with biy_runs_extend or run again.
• Decision: change the model in your code. Your agent can make that change when you ask; OptBench never changes your code by itself.
• Code changed since the run: biy_drift_check tells you which use cases no longer match what the code sends (Check before you merge).
• Going live: Connect traces so real requests become cases too.