Check before you merge

A benchmark measured the prompt, tools and model your code sent on the day it ran. Once a change edits a prompt, removes a tool or adds a new LLM call, that result may no longer describe your app. Before you merge, have your coding agent run biy_drift_check on the changed files.

What it checks

For each use case whose source_refs name a changed file:

• The prompt. A few short snippets of the stored prompt, looked for in the prompt's own file (or the call site, for an inline prompt).
• The tools. The tool names the use case sends, looked for where the tools are defined.
• The functions. The functions in source_refs, looked for in the call site.

Each of those use cases gets a state:

It also greps every changed file for LLM calls (SDK imports and calls, model names next to them) and lists the ones no use case covers. Tests are skipped.

Run it

Ask your agent:

Before I merge this branch, run OptBench's drift check on the changed files.

The agent compares the branch with where it merges, sends each changed file's text (null for a deleted file) and the commit, and reports each use case's state and the new LLM calls. With the Claude Code plugin, the optbench:check-drift skill does this, and Claude picks it when you're about to merge or open a PR.

OptBench reads the files to check them and keeps none of their text: only each use case's state, the paths, line numbers and the commit. Lockfiles, build output and other files that can't hold a prompt are left out.

What to do with the result

Your agent asks before each step; none of them is automatic.

• changed or missing. Refresh the use case from the new code: biy_scan_repo(part="refresh"). Your agent re-reads the prompt, tools and schema, and shows you a preview of what changes before it updates anything. Your cases, metrics and runs stay. A re-run comes next, with an estimate first. Runs measured on the old prompt are kept, marked as measured on an older prompt.
• A false alarm. If the refresh preview shows nothing changed, nothing is updated. Running the drift check again on the use case's files sets it back to ok.
• New LLM calls. Each one is a candidate for a new use case. Your agent can create it with a scan and offer to benchmark it before it ships (Benchmark before launch).

biy_use_cases_list shows drift: changed or missing on every use case that needs a refresh, until it is refreshed.

Limits

• It checks only what a use case's source_refs name. A prompt built at runtime from many places gives no snippets to look for, so its prompt reads as unknown, not as changed.
• A prompt counts as changed only when the scan recorded where it lives (a ref with the role prompt). For an inline prompt without one, an edited prompt reads as unknown; a removed tool or function still shows.
• It runs when your agent runs it. There is no check on push or on pull requests from OptBench's side.