Do it with your agent

Every screen in the dashboard has MCP tools that do the same thing, and both call the same code. This page goes screen by screen: what the screen is, which tools match it, and one prompt to try. You don't need the tool names in your prompt; they are here so you know what your agent will call.

Three things stay in the dashboard on purpose: entering secrets (provider keys, endpoint keys, the ingest key), setting your data rules, and managing members. Anything that costs money needs your yes to an estimate first.

Workspace

Which workspace your agent works in. With OAuth you can belong to several.

Tools: biy_workspaces_list, biy_workspaces_switch.

Which OptBench workspaces am I in? Switch to the staging one.

Settings

Trade-off

How much quality, speed and cost matter, as weights. Use cases and projects without their own weights use these.

Tools: biy_workspace_get, biy_workspace_update (default_profile). For one use case: biy_use_cases_update; for one project: biy_project_update.

Set our OptBench trade-off to quality 60, cost 30, speed 10.

Targets

Speed and cost score 1 at the target and 0 at the worst value you accept. Use cases without their own targets use these.

Tools: biy_workspace_get, biy_workspace_update (default_profile).

Show our OptBench speed and cost targets, and explain what a score of 0.5 means for each.

Models

The candidate models every run benchmarks, the fill-up pool, the judge jury, repeats and the per-run cost cap. Model rules decide which hosts may see your prompts (zero data retention, no training, regions, providers).

Tools: biy_models_list, biy_workspace_get, biy_workspace_update (candidate_pool, default_judge, default_repeats, max_run_cost_usd), biy_model_rules_get, biy_propose_model_rules. Your agent proposes model rules from what the code shows; you approve them in the dashboard. Your own endpoints are added in the dashboard, since they need a key.

Which models may see our prompts under our OptBench model rules? Add the cheapest one that supports tool calls to our candidate models.

Data rules

What your agent and OptBench may use from your data: which trace sources, how far back, how many cases per import, words that keep a record out, whether recorded answers and tool results are kept. You set them in Data → Rules; your agent can only read them.

Tools: biy_data_rules_get, biy_data_findings_report (what the agent found in a source, checked against your rules).

What do our OptBench data rules allow you to import from PostHog?

Members

Who is in the workspace and their role. Invites and roles are managed in the dashboard only. Your agent can tell you your own role.

Tools: biy_workspaces_list.

What is my role in our OptBench workspace?

Use cases

The list of every use case, with its case count and the latest verdict, and what each one costs per month today.

Tools: biy_use_cases_list, biy_cost_ranking.

List our OptBench use cases by monthly cost, and say which have never been benchmarked.

Setup

One use case's prompt, tools, output schema, metrics, gates, trade-off and today's model. A pipeline shows its steps instead of one prompt.

Tools: biy_use_cases_get, biy_use_cases_update, biy_use_cases_fill, biy_use_cases_check, biy_metrics_list, biy_use_cases_reselect_metrics, biy_upload_url (long prompts and schemas), biy_drift_check (does it still match the code).

Check the support chat use case in OptBench against the current code and fix what is out of date.

Cases

The inputs the models are tested on, where each came from, and whether it has a reference.

Tools: biy_cases_list, biy_cases_add, biy_cases_update, biy_cases_delete, biy_cases_generate, biy_cases_import, biy_upload_url.

Add the fixtures in tests/fixtures/invoices as cases for the invoice extraction use case, then generate cases up to 30.

Gold

What a correct answer must do, per case: checks and a final answer. Finals wait for your review. See Write gold.

Tools: biy_cases_list (without_gold=true), biy_cases_gold (free), biy_cases_gold_generate (costs money, estimate first).

Write gold for the support chat cases that have none, from the traces and our code.

Runs

Estimates, runs in progress, finished runs, and runs that are out of date because the prompt, the cases or the metrics changed.

Tools: biy_runs_estimate, biy_runs_estimate_many, biy_runs_start (needs confirm_cost_usd), biy_runs_status, biy_runs_extend, biy_runs_rescore.

Estimate a pilot and a full run for the summarizer use case. Don't start anything yet.

Verdict

The result of a run: Decision or Shortlist, the verdict ("Switch to A", "Keep today", "Use A", "Worth testing: A, B"), the evidence ("measured on N cases, k with a reference"), cost per successful task with its interval, and the tier reasons. See Benchmark before launch.

Tools: biy_runs_status, biy_use_cases_get (the latest run).

What did the last OptBench run for the summarizer find? Is it a Decision or a Shortlist, and what is missing for a Decision?

Data sources

Where your real data comes from: OpenTelemetry, PostHog, Langfuse, files, and provider bills. Each source shows whether your app is still tracking and how many cases came from it. See Connect traces.

Tools: biy_traces_list (what waits, with a suggested use case per group), biy_data_sources_pull (a direct source, preview then consent), biy_traces_import (staged OpenTelemetry traces, preview; you import them in the dashboard), biy_data_sources_list, biy_data_sources_register, biy_data_findings_report, biy_upload_url, biy_cases_import. Keys and the ingest key stay in the dashboard.

Find where this app sends its traces and import last week's support chat requests into OptBench. Show me the preview first.

Projects

A project is the repo a scan came from: its use cases, what it spends, its verdicts, and its own candidate models, trade-off and repeats (or the workspace's).

Tools: biy_project_get, biy_project_update, biy_use_cases_list (repo), biy_runs_estimate_many (repo), biy_scan_repo.

Show this repo's OptBench project settings, and estimate running all of its use cases.