# Tools

Every tool the OptBench MCP server exposes, with the first sentence of its description and its access:

- **read**: only reads
- **write**: creates or changes something
- **destructive**: deletes
- **spends**: runs paid model calls once you pass `confirm_cost_usd` from an estimate the user agreed to (without it, the tool only estimates)

Your client's `tools/list` is the source of truth; each tool's full description says how to call it. Start with `biy_scan_repo`.

<!-- tools:start -->
49 tools, generated from the server by `scripts/gen_mcp_docs.py`. Don't edit by hand.

### Start here
| Tool | Access | What it does |
|---|---|---|
| `biy_scan_repo` | read | Start here when the user asks to scan a repository with OptBench (find its LLM calls and benchmark them), or to refresh use cases whose code changed (drift changed / missing), or to use their real traces |

### Workspace and settings
| Tool | Access | What it does |
|---|---|---|
| `biy_project_get` | read | A project's settings (the repo a scan came from): the candidate models, trade-off profile and repeats its use cases run with, each the project's own or inherited from the workspace |
| `biy_project_update` | write | Set a project's own candidate models (candidate_pool.default_candidates), trade-off profile or repeats; a field sent as null inherits the workspace's again |
| `biy_workspace_get` | read | The company defaults that use cases and runs inherit: which models are benchmarked (the default candidates, compared with today's model of each use case), the jury, repeats, trade-off profile and the per-run cost cap |
| `biy_workspace_update` | write | Change company defaults that every use case and run inherits (any subset): default_profile, default_judge, default_references, candidate_pool, default_repeats, max_run_cost_usd |
| `biy_workspaces_list` | read | The workspaces you belong to, your role in each, and which one your calls use now |
| `biy_workspaces_switch` | write | Use another of your workspaces from now on: your agent's default until you switch again (the dashboard has its own switcher) |

### Models and rules
| Tool | Access | What it does |
|---|---|---|
| `biy_cost_ranking` | read | Free, no model calls: the workspace's use cases by estimated monthly expense today, highest first, from each use case's usage_estimate (or real volume) × today's model price |
| `biy_model_rules_get` | read | Which hosts may see this workspace's prompts: zero data retention, no training, allowed regions (e.g. EU only), allowed / blocked providers, OptBench-hosted models on or off; plus the scan's open proposals |
| `biy_models_list` | read | Benchmarkable models with VERIFIED capabilities (text, vision, tools, json_schema) and prices: open-weight, and (shortlist=false) the closed models OptBench knows, role "closed" |
| `biy_propose_model_rules` | write | Propose model rules from what the repo shows (scan_repo part "model_rules"): which providers and base URLs it calls (Azure OpenAI, Bedrock, Vertex, a self-hosted vLLM), region settings, data retention / no-training / ZDR settings or comments, timeouts or latency budgets |

### Your data
| Tool | Access | What it does |
|---|---|---|
| `biy_data_findings_report` | write | Report what you found in a trace source BEFORE exporting anything, so the person sees it (Data → Rules) and OptBench checks each item against their data rules |
| `biy_data_rules_get` | read | What the person allows you to use from their data: which sources (PostHog, Langfuse, OpenTelemetry, files), how far back, how many cases, which words keep a record out, whether recorded answers and tool results are kept, whether Jev may read sample requests |
| `biy_data_sources_list` | read | Where this workspace's imported real data comes from (PostHog, Langfuse, OTel, files): mode, host, project, imported case count, last import |
| `biy_data_sources_pull` | write | Fetch the newest traces of a direct PostHog / Langfuse source with the key the dashboard stores, as cases of a use case: the same as the dashboard's Pull |
| `biy_data_sources_register` | write | Record where you export the user's traces from, BEFORE importing them, so the dashboard (Data → Sources) shows the source and can delete its cases together |

### Real traffic and the lifecycle
| Tool | Access | What it does |
|---|---|---|
| `biy_coverage_get` | read | Do the use case's own cases (synthetic, from the repo, written by hand, uploaded) cover what production asks (lifecycle entry C)? Production = its imported trace cases (source "imported", default) or the staged OpenTelemetry traces routed to it (source "otel") |
| `biy_lifecycle_get` | read | Where a use case stands in the lifecycle, and what to do next with which tools |
| `biy_traces_import` | read | Preview staged OpenTelemetry traces as cases of a use case: mapped and redacted in memory, nothing stored |
| `biy_traces_list` | read | Start here for real traffic (lifecycle entry B or C): what traces wait to become cases, grouped by agent or span name, with the use case each group most likely belongs to and why (at least half of the tools called must be the use case's; without tools, the agent's name) |
| `biy_traces_route` | write | The person says which use case an unclear or unrouted group of staged traces belongs to (from biy_traces_list: its name, service and prompt fingerprint) |

### Metrics
| Tool | Access | What it does |
|---|---|---|
| `biy_metrics_list` | read | Metrics from the metric bank |
| `biy_metrics_select` | read | Pick metrics + gates for ONE use case from what you saw in the code |

### Use cases
| Tool | Access | What it does |
|---|---|---|
| `biy_drift_check` | write | After a code change: are the repo's use cases still what the code sends, and does the change add LLM calls no use case covers? files: the changed files as {path: full text} (null for a deleted file), e.g. from `git diff --name-only` |
| `biy_use_cases_check` | read | Check use cases before estimating (scan_repo part "use_case" after filling each one, step 6 for the repo) |
| `biy_use_cases_create` | write | Create use cases |
| `biy_use_cases_delete` | destructive | Delete use cases with their cases and runs (e.g. duplicates from an interrupted scan) |
| `biy_use_cases_fill` | write | Fill in ONE use case in a single call (scan_repo part "use_case"), instead of metrics_select + update + cases_add |
| `biy_use_cases_get` | read | One use case in full (io_spec with prompts and schema, metrics, gates, profile, judge), and its background case generation when one runs or failed |
| `biy_use_cases_list` | read | Use cases in the workspace, with case counts and the latest run's recommendation |
| `biy_use_cases_reselect_metrics` | write | Run the metric selector again for an existing use case and show the diff (added / removed metrics and gates) |
| `biy_use_cases_update` | write | Change part of a use case, e.g. the trade-off: {"profile": {"weights": {"quality": 50, "speed": 10, "cost": 40}}} |

### Cases and gold
| Tool | Access | What it does |
|---|---|---|
| `biy_cases_add` | write | Add test cases |
| `biy_cases_delete` | destructive | Delete cases from a use case (ids from biy_cases_list) |
| `biy_cases_generate` | write | Generate synthetic test cases, seeded from the use case's existing cases, in the background: it returns at once and OptBench writes the cases in the background |
| `biy_cases_gold` | write | Write gold onto cases: what a correct answer must do, so a benchmark scores right answers, not agreement with today's model |
| `biy_cases_gold_generate` | read, spends | Only when the user asks OptBench to write the gold instead of you (biy_cases_gold is free: prefer it) |
| `biy_cases_import` | write | Turn an export of the user's OWN data into test cases, redacted: realistic inputs, and for agent use cases replay tool mocks from the recorded tool calls |
| `biy_cases_list` | read | Cases of a use case, compact: id, a short preview of input and reference, source, tags |
| `biy_cases_update` | write | Edit one case: any of input, reference (null clears it), tags |
| `biy_upload_bundle_url` | write | One upload for many use cases at once: prompts, schemas, tool lists, stores and cases of a whole batch in one file and one curl, instead of a biy_upload_url and a curl per field |
| `biy_upload_url` | write | One-time URLs to upload large fields straight from files, so their bytes never go through your tool arguments |

### Runs and results
| Tool | Access | What it does |
|---|---|---|
| `biy_runs_estimate` | read | Free: computes the cost and call count of a run, calls no model and spends nothing |
| `biy_runs_estimate_many` | read | Free: the estimate for many use cases in one call (scan_repo part "estimate"): each with the workspace's default candidates and references, one compact line per use case and the total |
| `biy_runs_extend` | write, spends | Grow a finished run instead of running it again: add models, open or closed (only they run, on the run's own cases and repeats, and only their answers are judged), retry the calls that failed (retry_errors: an expired key, a timeout; only those calls run again and their answers are judged by the run's jury), or set a new jury (every stored answer is judged again, no model calls) |
| `biy_runs_rescore` | write | Score a finished run's stored outputs again with today's scorers and the use case's current metrics (e.g. after biy_use_cases_reselect_metrics) |
| `biy_runs_start` | write, spends | Start a benchmark run |
| `biy_runs_status` | read | Progress, leaderboard so far, and the recommendation once done |

### Other
| Tool | Access | What it does |
|---|---|---|
| `biy_traffic_get` | read | Real traffic as the production loop counts it from the traces that reach OptBench (every 10 minutes): with use_case_id, that use case per day and per model (traces, model and tool calls, tokens, the cost the traces recorded, distinct requests among a sample); without, each use case's traces and spend and the traffic no use case claims |
<!-- tools:end -->