# Do it with your agent

Every screen in the dashboard has MCP tools that do the same thing, and both call the same code. This page goes screen by screen: what the screen is, which tools match it, and one prompt to try. You don't need the tool names in your prompt; they are here so you know what your agent will call.

Three things stay in the dashboard on purpose: entering secrets (provider keys, endpoint keys, the ingest key), setting your data rules, and managing members. Anything that costs money needs your yes to an estimate first.

## Workspace

Which workspace your agent works in. With OAuth you can belong to several.

Tools: `biy_workspaces_list`, `biy_workspaces_switch`.

```text
Which OptBench workspaces am I in? Switch to the staging one.
```

## Settings

### Trade-off

How much quality, speed and cost matter, as weights. Use cases and projects without their own weights use these.

Tools: `biy_workspace_get`, `biy_workspace_update` (`default_profile`). For one use case: `biy_use_cases_update`; for one project: `biy_project_update`.

```text
Set our OptBench trade-off to quality 60, cost 30, speed 10.
```

### Targets

Speed and cost score 1 at the target and 0 at the worst value you accept. Use cases without their own targets use these.

Tools: `biy_workspace_get`, `biy_workspace_update` (`default_profile`).

```text
Show our OptBench speed and cost targets, and explain what a score of 0.5 means for each.
```

### Models

The candidate models every run benchmarks, the fill-up pool, the judge jury, repeats and the per-run cost cap. Model rules decide which hosts may see your prompts (zero data retention, no training, regions, providers).

Tools: `biy_models_list`, `biy_workspace_get`, `biy_workspace_update` (`candidate_pool`, `default_judge`, `default_repeats`, `max_run_cost_usd`), `biy_model_rules_get`, `biy_propose_model_rules`. Your agent proposes model rules from what the code shows; you approve them in the dashboard. Your own endpoints are added in the dashboard, since they need a key.

```text
Which models may see our prompts under our OptBench model rules? Add the cheapest one that supports tool calls to our candidate models.
```

### Data rules

What your agent and OptBench may use from your data: which trace sources, how far back, how many cases per import, words that keep a record out, whether recorded answers and tool results are kept. You set them in **Data → Rules**; your agent can only read them.

Tools: `biy_data_rules_get`, `biy_data_findings_report` (what the agent found in a source, checked against your rules).

```text
What do our OptBench data rules allow you to import from PostHog?
```

### Members

Who is in the workspace and their role. Invites and roles are managed in the dashboard only. Your agent can tell you your own role.

Tools: `biy_workspaces_list`.

```text
What is my role in our OptBench workspace?
```

## Use cases

The list of every use case, with its case count and the latest verdict, and what each one costs per month today.

Tools: `biy_use_cases_list`, `biy_cost_ranking`.

```text
List our OptBench use cases by monthly cost, and say which have never been benchmarked.
```

### Setup

One use case's prompt, tools, output schema, metrics, gates, trade-off and today's model. A pipeline shows its steps instead of one prompt.

Tools: `biy_use_cases_get`, `biy_use_cases_update`, `biy_use_cases_fill`, `biy_use_cases_check`, `biy_metrics_list`, `biy_use_cases_reselect_metrics`, `biy_upload_url` (long prompts and schemas), `biy_drift_check` (does it still match the code).

```text
Check the support chat use case in OptBench against the current code and fix what is out of date.
```

### Cases

The inputs the models are tested on, where each came from, and whether it has a reference.

Tools: `biy_cases_list`, `biy_cases_add`, `biy_cases_update`, `biy_cases_delete`, `biy_cases_generate`, `biy_cases_import`, `biy_upload_url`.

```text
Add the fixtures in tests/fixtures/invoices as cases for the invoice extraction use case, then generate cases up to 30.
```

### Gold

What a correct answer must do, per case: checks and a final answer. Finals wait for your review. See [Write gold](/docs/workflow-gold).

Tools: `biy_cases_list` (`without_gold=true`), `biy_cases_gold` (free), `biy_cases_gold_generate` (costs money, estimate first).

```text
Write gold for the support chat cases that have none, from the traces and our code.
```

### Runs

Estimates, runs in progress, finished runs, and runs that are out of date because the prompt, the cases or the metrics changed.

Tools: `biy_runs_estimate`, `biy_runs_estimate_many`, `biy_runs_start` (needs `confirm_cost_usd`), `biy_runs_status`, `biy_runs_extend`, `biy_runs_rescore`.

```text
Estimate a pilot and a full run for the summarizer use case. Don't start anything yet.
```

### Verdict

The result of a run: Decision or Shortlist, the verdict ("Switch to A", "Keep today", "Use A", "Worth testing: A, B"), the evidence ("measured on N cases, k with a reference"), cost per successful task with its interval, and the tier reasons. See [Benchmark before launch](/docs/workflow-before-launch).

Tools: `biy_runs_status`, `biy_use_cases_get` (the latest run).

```text
What did the last OptBench run for the summarizer find? Is it a Decision or a Shortlist, and what is missing for a Decision?
```

## Data sources

Where your real data comes from: OpenTelemetry, PostHog, Langfuse, files, and provider bills. Each source shows whether your app is still tracking and how many cases came from it. See [Connect traces](/docs/workflow-connect-traces).

Tools: `biy_traces_list` (what waits, with a suggested use case per group), `biy_data_sources_pull` (a direct source, preview then consent), `biy_traces_import` (staged OpenTelemetry traces, preview; you import them in the dashboard), `biy_data_sources_list`, `biy_data_sources_register`, `biy_data_findings_report`, `biy_upload_url`, `biy_cases_import`. Keys and the ingest key stay in the dashboard.

```text
Find where this app sends its traces and import last week's support chat requests into OptBench. Show me the preview first.
```

## Projects

A project is the repo a scan came from: its use cases, what it spends, its verdicts, and its own candidate models, trade-off and repeats (or the workspace's).

Tools: `biy_project_get`, `biy_project_update`, `biy_use_cases_list` (`repo`), `biy_runs_estimate_many` (`repo`), `biy_scan_repo`.

```text
Show this repo's OptBench project settings, and estimate running all of its use cases.
```