How OptBench treats your data

OptBench tests models on cases from your own product, and some of those cases can come from your real traffic. This page explains what happens to that data on its way from your tools to the models that answer it. It describes what OptBench does today.

Three ways in

Your traces live in PostHog, Langfuse, an OpenTelemetry backend or a file. OptBench gets them in one of three ways.

Through your coding agent. Claude Code or Cursor, connected to OptBench, runs the export on your machine with your own keys and uploads the result. OptBench never receives those keys.
A direct connection. OptBench reads from the tool itself, when you test the connection or pull traces. Connect with PostHog signs you in at PostHog and asks for two scopes only, query:read and project:read. OptBench stores that connection encrypted in Supabase Vault, refreshes it when it is used, and revokes it at PostHog when you remove the source. Or you paste a key: a PostHog personal API key with only query:read, or a Langfuse project key pair. Langfuse keys can also write; OptBench only sends read requests with them. A pasted key is stored encrypted in Supabase Vault too, and deleted when you remove the source.
OpenTelemetry push. Your Collector or app sends a copy of its spans to OptBench with an ingest key that can do nothing else in OptBench. OptBench keeps only GenAI spans, redacts them as they arrive, before anything is written, and keeps them for a day for you to preview. You choose how many traces it keeps (a sample rate), and you can revoke the key at any time. Nothing becomes a test case until you import it.

Files come in through your agent, or you upload them yourself on a use case.

You decide what may be used

Connecting a source in the dashboard allows it, in the same step. The Rules & privacy tab of the Data page lists the allowed sources and holds the rest of your rules. Only you set them, from a signed-in session in the dashboard. A source your agent registers stays blocked until you allow it. Your agent can read them but not change them: no OptBench tool for agents sets them, and a request made with an agent key is refused.

A project can have its own data rules (project page → Production), but they can only narrow the workspace's, never widen them. An agent working in a repo reads that project's rules (biy_data_rules_get(project=…)), and a source it registers from there names the repo it feeds (repos), so the source only feeds that project.

Until you save your rules, only files you upload are allowed. PostHog, Langfuse and OpenTelemetry stay off. The rules cover:

which sources may be used
whether recorded answers are kept
whether recorded tool results are kept
how far back an import may reach, in days
how many cases one import may bring in
words that keep a record out: a record that contains one of them is not imported

Your agent asks before it exports

Before exporting anything, your agent reports what it found: names, counts and field names, never the content. OptBench marks each item allowed or not by your rules. The agent then has to show you the list and ask which ones to use.

What happens on every import

An import can start in the dashboard, as a direct pull, or from your agent. Each one is checked on the server, in this order:

A source that is off in your rules is refused before anything is read. A direct pull from a source that is off never contacts the host.
Redaction always runs before anything is stored. Emails, phone numbers, IBANs, card numbers, IP addresses, tokens in URLs, API keys and long numeric IDs become placeholders like <EMAIL_1>. UUIDs and ids with a prefix (cust_48213, ORD-2026-00017) become <INTERNAL_ID_1> too, unless your data rules turn that off. Names and street addresses are not found.
You see a preview first: the cases, what was redacted and any warnings. Nothing is imported until you explicitly agree.

In your traceHi, it's Lena Fischer (lena.fischer@example.com). Call me back on +49 151 23456789, order 4471029384.

What OptBench storesHi, it's Lena Fischer (<EMAIL_1>). Call me back on <PHONE_1>, order <ID_1>.

Redaction matches patterns. It also replaces UUIDs and ids with a prefix, such as cust_48213 or ORD-2026-00017 (on unless your data rules turn it off). It doesn't find names, street addresses or other free text, which is why the name stays. If something must never come in, add it to the words that keep a record out.

What OptBench keeps, and for how long

An export waits in a staging slot for at most 30 minutes, then it is deleted.
OpenTelemetry spans, already redacted, wait for a day after their trace's last span, then they are deleted. You can clear them sooner.
Imported cases are kept until you delete them, or the use case they belong to.
Each import is logged: who ran it, when, from which source and how many cases it brought in.
You can delete one import on its own, all cases from one source, or all imported data.

What OptBench stores, at the bottom of the Data tab, lists everything kept for your workspace and how long it stays.

Which models see your cases

The candidate models and your current model answer your test cases. They run through OpenRouter on OptBench's account, by default only on hosts that keep no prompts or answers (zero data retention), or through an endpoint you added, on your own account.
Judge models read the answers to score them. By default that is a decision model, Jev 1.14, with DeepSeek-V4.1-Flash as a fallback for answers Jev can't score, both on hosts that keep no prompts or answers.
When you press Draft with AI, your workspace's gold model reads a case's conversation, its recorded tool results and the prompt to propose the right answer. By default that is GPT-6.1 Sol, through OpenRouter on a host that keeps no prompts or answers; you can pick another in Settings. Nothing is sent until you confirm the cost.
If your model rules keep everything on private hosts, the decision models are left out and an open-weight judge, DeepSeek-V4.1-Flash, scores the answers instead.

What OptBench never stores

The model-provider keys in your code. A key you add yourself, for your own endpoint or a connected source, is the exception: it is kept encrypted in Supabase Vault and never shown again.
Your PostHog, Langfuse or OpenTelemetry keys when your agent does the export.
Raw, unredacted exports in its database.

How we measure this site

OptBench uses PostHog (EU) to count page views and clicks. Session replay runs only on the public pages (this one, the landing page, the docs and sign-in), with form fields masked. In the dashboard there is no replay and clicks are counted without the text of what you clicked, so your traces, cases and keys never reach it.

Set or change your rules on the Rules & privacy tab of the Data page (sign in first).