# Monitor in production

A benchmark picks a model on the day you run it. In production your traffic shifts, prompts change and prices move. This page is about keeping the choice right after launch.

This is where OptBench is going, and only part of it is built. Each section says what you can use **today** and what is **coming**. Nothing marked coming exists yet.

## Volume and cost per use case

**Today.**

- **Provider bills.** In **Data → Sources**, add OpenAI usage, Anthropic usage or OpenRouter activity: upload the console's CSV export, or connect an admin key. These sources carry volume and cost only, never a prompt or an answer. Then assign each bill line (model, project, key) to the use cases it pays for. The dashboard shows where your money goes, and what isn't matched to a use case yet.
- **Volume on a use case.** A use case can carry real calls and spend per month, with where the number comes from. Your agent sets it from real numbers only (`biy_use_cases_update`).
- **Cost ranking.** `biy_cost_ranking` lists your use cases by estimated monthly cost today, highest first, so you benchmark the expensive ones first. Free, no model calls.
- **Tracking.** Each trace source shows whether your app still sends LLM calls, how recently, and how many carry a prompt and an answer.

**Coming.** Volume and cost per use case from the traces themselves (OpenTelemetry token counts and cost), with no content needed, so it works under strict data rules. Then a "go live" checklist at the end of a benchmark that sets this up.

## A quality sample of live traces

**Today.** You can bring real requests into a use case's cases ([Connect traces](/docs/workflow-connect-traces)), write gold for them ([Write gold](/docs/workflow-gold)) and benchmark again. **Refresh sample** adds only the requests the use case doesn't have yet.

**Coming.** A small sample of live traces (by default 1–5%, capped per day) scored against the use case's own gates, as a rolling pass rate per gate, flagging guardrail breaches and drift. Off by default, and bound by your data and model rules. Your own coding agent will be able to score the sample over MCP instead, so the judge cost and the data stay with you.

## Proven savings after a switch

**Today.** A verdict states the cost per successful task of the verdict's model, with its 95% interval, against today's model. That is a prediction.

**Coming.** After you switch, OptBench compares cost per request before and after, on the same mix of requests, and shows the predicted saving next to the realized one, with a range and the sample size it needs. When you have a gateway, it suggests a canary (5–10% of traffic) first.

## Coverage of production by your cases

**Today.** Over MCP, `biy_coverage_get(use_case_id)` compares your own cases (synthetic, from the repo, written or uploaded) with the requests of the use case's imported traces. It clusters them locally on their opening words (no model call, nothing leaves OptBench) and says which clusters your cases cover. Below 30 production requests it tells you that's too few to act on. You still choose which traces become cases. The report in the dashboard is coming.

**Coming.** OptBench groups your production requests into clusters and maps each test case to its nearest cluster. You get a report such as "your cases cover 7 of 9 production clusters; add these 5; these 3 look stale", and new cases picked from the clusters you don't cover.

## Code changes

**Today.**

- `biy_drift_check` tells you, after a code change, whether each use case still matches what the code sends, and whether the change adds LLM calls no use case covers ([Check before you merge](/docs/workflow-drift)).
- A use case flags runs measured on another prompt, on cases that changed since, or with other metrics. `biy_runs_rescore` scores stored answers again with today's metrics, without new model calls.

## Better estimates

**Coming.** Run estimates priced from the real lengths of your production answers and tool calls (p50 and p90), not from a pilot's average. Each run will make the next estimate more accurate.

## The order we are building it

- First: volume and cost per use case from traces, and the go-live checklist.
- Then: proven savings after a switch.
- Then: estimates from real production lengths.
- Then: the coverage report.
- Last: the quality sample of live traces.