Example prompts

Things to ask your coding agent once OptBench is connected. The agent picks the tools; you don't need their names.

Start

"Scan this repo with OptBench." Finds the LLM calls and sets up a use case for each, with metrics and test cases. In Claude Code you can also run /mcp__optbench__scan_repo.
"Which LLM calls in this repo cost the most? Use OptBench's cost ranking." Free, no model calls.

"Estimate a benchmark run for the invoice extraction use case." Free. Shows the models, cases and the cost.
"Start that run." The agent asks you to confirm the cost first.
"How is the run going? Which model does OptBench recommend?"
"Add Qwen and DeepSeek to the finished run instead of starting over."

"Add the fixtures in tests/fixtures/support as test cases for the support chat use case."
"Generate 20 more test cases for the summarizer."
"Import last week's traces from PostHog as test cases." The agent checks your data rules first and shows a preview before importing.
"Write the expected answers for the imported cases that have none."

"Before we launch, benchmark the new summarizer: write 3 cases with me, generate the rest up to 30, then estimate a pilot and the full run." See Benchmark before launch.
"Add OptBench as a second OpenTelemetry exporter in this app. Read the ingest key from OPTBENCH_INGEST_KEY and keep our existing exporter." See Connect traces.
"Refresh the support chat sample from PostHog and tell me how many requests are new." The agent shows the preview before importing.
"Write gold for the imported support chat cases from the traces and our code. Checks only, no final answers." Free. See Write gold.
"Estimate what it costs to have OptBench draft gold for the support chat cases that have none. Don't start it." Shows the cost; nothing runs until you confirm.
"Is the last summarizer run a Decision or a Shortlist? What would it take to get a Decision?" For what comes after launch, see Monitor in production.