# OptBench

> OptBench finds the cheapest model that clears your quality bar for each AI call in a repo. It tests other models, open or closed, against the one each call uses today, on your own cases, and says to switch, to keep today's model, or which cases or approved references it still needs. Coding agents use it over MCP: scan a repo, set up use cases with metrics and test cases, estimate and run benchmarks, read the verdict.

MCP server: https://app.optbench.com/mcp (Streamable HTTP). Auth: OAuth 2.1, or `Authorization: Bearer <workspace key>` (keys start with `optbench_`).
Claude Code: `claude mcp add --transport http optbench https://app.optbench.com/mcp`, then `/mcp` → Authenticate.
To start, call `biy_scan_repo` and follow the instructions it returns. Runs cost money: `biy_runs_start` needs `confirm_cost_usd` from `biy_runs_estimate`, after the user agrees.

## Methodology

- [How OptBench decides](https://optbench.com/methodology.md): use cases per AI call, gold references, judging with Jev, the paired bootstrap and Holm test against today's model, the 30-case and 50%-gold floors, Shortlist vs Decision, and what a verdict does not claim

## Docs

- [Overview](https://optbench.com/docs/index.md): what OptBench does for a coding agent, and one-step setup
- [Clients](https://optbench.com/docs/clients.md): setup for Claude Code, Cursor, Codex, VS Code and any MCP client
- [MCP server](https://optbench.com/docs/mcp-server.md): endpoint, OAuth vs workspace key, what an agent may do, spending guard
- [Tools](https://optbench.com/docs/tools.md): every tool with its access (read / write / destructive / spends), generated from the server
- [Example prompts](https://optbench.com/docs/prompts.md): what to ask your agent

## Workflows

- [Benchmark before launch](https://optbench.com/docs/workflow-before-launch.md): scan, one use case per call site, hand and generated cases, pilot, estimate, run; Shortlist vs Decision
- [Connect traces](https://optbench.com/docs/workflow-connect-traces.md): data rules, OpenTelemetry push, PostHog and Langfuse pulls, preview, Refresh sample
- [Write gold](https://optbench.com/docs/workflow-gold.md): checks and finals; your agent (free), Draft with AI (estimate first), or by hand; review
- [Check before you merge](https://optbench.com/docs/workflow-drift.md): biy_drift_check on the changed files; ok / changed / missing per use case, new LLM calls no use case covers, refresh and re-run
- [Monitor in production](https://optbench.com/docs/workflow-production.md): what exists today after launch and what is coming
- [Do it with your agent](https://optbench.com/docs/with-your-agent.md): each dashboard screen, the tools that do the same, one prompt each

## Optional

- [Full text](https://optbench.com/llms-full.txt): the methodology, every docs page and the research articles in one file
- [Your data](https://optbench.com/your-data): how OptBench treats your data