Tools

Every tool the OptBench MCP server exposes, with the first sentence of its description and its access:

read: only reads
write: creates or changes something
destructive: deletes
spends: runs paid model calls once you pass confirm_cost_usd from an estimate the user agreed to (without it, the tool only estimates)

Your client's tools/list is the source of truth; each tool's full description says how to call it. Start with biy_scan_repo.

49 tools, generated from the server by scripts/gen_mcp_docs.py. Don't edit by hand.

Start here

Tool	Access	What it does
`biy_scan_repo`	read	Start here when the user asks to scan a repository with OptBench (find its LLM calls and benchmark them), or to refresh use cases whose code changed (drift changed / missing), or to use their real traces

Workspace and settings

Tool	Access	What it does
`biy_project_get`	read	A project's settings (the repo a scan came from): the candidate models, trade-off profile and repeats its use cases run with, each the project's own or inherited from the workspace
`biy_project_update`	write	Set a project's own candidate models (candidate_pool.default_candidates), trade-off profile or repeats; a field sent as null inherits the workspace's again
`biy_workspace_get`	read	The company defaults that use cases and runs inherit: which models are benchmarked (the default candidates, compared with today's model of each use case), the jury, repeats, trade-off profile and the per-run cost cap
`biy_workspace_update`	write	Change company defaults that every use case and run inherits (any subset): default_profile, default_judge, default_references, candidate_pool, default_repeats, max_run_cost_usd
`biy_workspaces_list`	read	The workspaces you belong to, your role in each, and which one your calls use now
`biy_workspaces_switch`	write	Use another of your workspaces from now on: your agent's default until you switch again (the dashboard has its own switcher)

Models and rules

Tool	Access	What it does
`biy_cost_ranking`	read	Free, no model calls: the workspace's use cases by estimated monthly expense today, highest first, from each use case's usage_estimate (or real volume) × today's model price
`biy_model_rules_get`	read	Which hosts may see this workspace's prompts: zero data retention, no training, allowed regions (e.g. EU only), allowed / blocked providers, OptBench-hosted models on or off; plus the scan's open proposals
`biy_models_list`	read	Benchmarkable models with VERIFIED capabilities (text, vision, tools, json_schema) and prices: open-weight, and (shortlist=false) the closed models OptBench knows, role "closed"
`biy_propose_model_rules`	write	Propose model rules from what the repo shows (scan_repo part "model_rules"): which providers and base URLs it calls (Azure OpenAI, Bedrock, Vertex, a self-hosted vLLM), region settings, data retention / no-training / ZDR settings or comments, timeouts or latency budgets

Your data

Tool	Access	What it does
`biy_data_findings_report`	write	Report what you found in a trace source BEFORE exporting anything, so the person sees it (Data → Rules) and OptBench checks each item against their data rules
`biy_data_rules_get`	read	What the person allows you to use from their data: which sources (PostHog, Langfuse, OpenTelemetry, files), how far back, how many cases, which words keep a record out, whether recorded answers and tool results are kept, whether Jev may read sample requests
`biy_data_sources_list`	read	Where this workspace's imported real data comes from (PostHog, Langfuse, OTel, files): mode, host, project, imported case count, last import
`biy_data_sources_pull`	write	Fetch the newest traces of a direct PostHog / Langfuse source with the key the dashboard stores, as cases of a use case: the same as the dashboard's Pull
`biy_data_sources_register`	write	Record where you export the user's traces from, BEFORE importing them, so the dashboard (Data → Sources) shows the source and can delete its cases together

Real traffic and the lifecycle

Tool	Access	What it does
`biy_coverage_get`	read	Do the use case's own cases (synthetic, from the repo, written by hand, uploaded) cover what production asks (lifecycle entry C)? Production = its imported trace cases (source "imported", default) or the staged OpenTelemetry traces routed to it (source "otel")
`biy_lifecycle_get`	read	Where a use case stands in the lifecycle, and what to do next with which tools
`biy_traces_import`	read	Preview staged OpenTelemetry traces as cases of a use case: mapped and redacted in memory, nothing stored
`biy_traces_list`	read	Start here for real traffic (lifecycle entry B or C): what traces wait to become cases, grouped by agent or span name, with the use case each group most likely belongs to and why (at least half of the tools called must be the use case's; without tools, the agent's name)
`biy_traces_route`	write	The person says which use case an unclear or unrouted group of staged traces belongs to (from biy_traces_list: its name, service and prompt fingerprint)

Metrics

Tool	Access	What it does
`biy_metrics_list`	read	Metrics from the metric bank
`biy_metrics_select`	read	Pick metrics + gates for ONE use case from what you saw in the code

Use cases

Tool	Access	What it does
`biy_drift_check`	write	After a code change: are the repo's use cases still what the code sends, and does the change add LLM calls no use case covers? files: the changed files as {path: full text} (null for a deleted file), e.g. from `git diff --name-only`
`biy_use_cases_check`	read	Check use cases before estimating (scan_repo part "use_case" after filling each one, step 6 for the repo)
`biy_use_cases_create`	write	Create use cases
`biy_use_cases_delete`	destructive	Delete use cases with their cases and runs (e.g. duplicates from an interrupted scan)
`biy_use_cases_fill`	write	Fill in ONE use case in a single call (scan_repo part "use_case"), instead of metrics_select + update + cases_add
`biy_use_cases_get`	read	One use case in full (io_spec with prompts and schema, metrics, gates, profile, judge), and its background case generation when one runs or failed
`biy_use_cases_list`	read	Use cases in the workspace, with case counts and the latest run's recommendation
`biy_use_cases_reselect_metrics`	write	Run the metric selector again for an existing use case and show the diff (added / removed metrics and gates)
`biy_use_cases_update`	write	Change part of a use case, e.g. the trade-off: {"profile": {"weights": {"quality": 50, "speed": 10, "cost": 40}}}

Cases and gold

Tool	Access	What it does
`biy_cases_add`	write	Add test cases
`biy_cases_delete`	destructive	Delete cases from a use case (ids from biy_cases_list)
`biy_cases_generate`	write	Generate synthetic test cases, seeded from the use case's existing cases, in the background: it returns at once and OptBench writes the cases in the background
`biy_cases_gold`	write	Write gold onto cases: what a correct answer must do, so a benchmark scores right answers, not agreement with today's model
`biy_cases_gold_generate`	readspends	Only when the user asks OptBench to write the gold instead of you (biy_cases_gold is free: prefer it)
`biy_cases_import`	write	Turn an export of the user's OWN data into test cases, redacted: realistic inputs, and for agent use cases replay tool mocks from the recorded tool calls
`biy_cases_list`	read	Cases of a use case, compact: id, a short preview of input and reference, source, tags
`biy_cases_update`	write	Edit one case: any of input, reference (null clears it), tags
`biy_upload_bundle_url`	write	One upload for many use cases at once: prompts, schemas, tool lists, stores and cases of a whole batch in one file and one curl, instead of a biy_upload_url and a curl per field
`biy_upload_url`	write	One-time URLs to upload large fields straight from files, so their bytes never go through your tool arguments

Runs and results

Tool	Access	What it does
`biy_runs_estimate`	read	Free: computes the cost and call count of a run, calls no model and spends nothing
`biy_runs_estimate_many`	read	Free: the estimate for many use cases in one call (scan_repo part "estimate"): each with the workspace's default candidates and references, one compact line per use case and the total
`biy_runs_extend`	writespends	Grow a finished run instead of running it again: add models, open or closed (only they run, on the run's own cases and repeats, and only their answers are judged), retry the calls that failed (retry_errors: an expired key, a timeout; only those calls run again and their answers are judged by the run's jury), or set a new jury (every stored answer is judged again, no model calls)
`biy_runs_rescore`	write	Score a finished run's stored outputs again with today's scorers and the use case's current metrics (e.g. after biy_use_cases_reselect_metrics)
`biy_runs_start`	writespends	Start a benchmark run
`biy_runs_status`	read	Progress, leaderboard so far, and the recommendation once done

Other

Tool	Access	What it does
`biy_traffic_get`	read	Real traffic as the production loop counts it from the traces that reach OptBench (every 10 minutes): with use_case_id, that use case per day and per model (traces, model and tool calls, tokens, the cost the traces recorded, distinct requests among a sample); without, each use case's traces and spend and the traffic no use case claims