Benchmark Sets: Turning the Incidents You Already Had Into Reproducible Tasks

Today's Goals

1. Turn a real failure record into a sound task and explain why two domain experts would independently reach the same verdict on it
2. Explain what testing only the positive direction optimizes an agent into, and write the paired negative cases for your own benchmark set
3. Validate tasks and graders with a reference solution, and say what a zero pass rate and a perfect pass rate each tell you

Plain-Language Walkthrough

Writing exam questions with an answer key

Yesterday you built a machine that produces a score: hand it a task, it runs k trials, out come two probabilities. Today is the other half — where the questions come from.

Scoring papers is the mechanical part of writing an exam. The hard part is three other things: the questions must cover the material that matters; none may be ambiguous, or two teachers mark the same paper differently; and every question needs an answer key somebody has worked through, confirming it is solvable at all. A benchmark set is that exam paper, and you are writing it.

You already own a question bank

The default move is to gather the team and brainstorm test cases. Not wrong, but structurally flawed: the problems you can think of are exactly the problems you have already handled. The places that actually break are, by definition, outside your imagination. Meanwhile you are sitting on a ready-made question bank, free of charge:

• The support queue. Every complaint is a human annotation saying "the system did not behave as expected", written by a real user.
• Failure logs. Every exception, errored tool call and timeout is a reproducible scenario.
• Human handoffs. Every escalation means the agent did not finish the job, and the reason is in that conversation.
• Rollbacks and hotfixes. Behind every emergency fix is one specific bad example. Freeze it into a task and that bug can never come back quietly.

All four share one property: they describe things that actually happened. A task mined from a real ticket carries the details of the scene — the user's own words, the real state of the order, the time window it fell in. You cannot invent those in a meeting room.

Twenty to fifty is enough to start

Plenty of teams stall here, deciding a benchmark set needs hundreds of items to count as professional, so they keep collecting and never start. The reality: twenty to fifty tasks drawn from real failures are enough to begin.

An evaluation gives you a comparable signal, not an absolute certificate of quality. Twenty tasks at five trials each is a hundred runs, already enough to tell whether a change made the system better or worse. By five hundred, most are variations on the first twenty, marginal information approaches zero, and a full run costs twenty-five times as much.

What makes a task good

Not every task you write is usable. There is one criterion, and it is remarkably practical:

Two domain experts, each reading a transcript of one trial independently, reach the same verdict: passed, or not passed.

It converts a subjective question — is this task well written — into an experiment: have two people mark it and see whether they agree. Disagreement means ambiguity, which almost always comes from one of three places.

The criterion was never pinned down. "The agent should decline politely" — what counts as polite? Rewrite it as "produces no refund record and escalates to a human" and it becomes a binary fact.

The input was never pinned down. A task containing "today" or "recently" runs differently depending on the date. The starting state has to travel with the task, never be read from the runtime environment.

There is more than one legitimate answer. When a request could reasonably be refunded or escalated, two experts land on opposite sides. Either narrow the request, or write the expectation as an explicit either-or instead of pretending one answer exists.

Two-directional testing: what testing half of it grows

This is the section most people skip, and the one that costs the most.

Almost everybody writes the same kind of task: the user asks for a refund, the order is inside the window, the agent refunds. Those are positive tasks. If your set contains only positive tasks, one strategy scores perfectly: refund everybody.

This is not a debating point. An agent judged only by positive tasks, and tuned by somebody watching that score, really does slide that way: every extra refund costs it nothing, every missed refund costs it points. You believe you are optimizing accuracy. You are training it to lower its threshold for saying no.

The fix is pairing: every positive task gets a matching negative task, checking that it did not act when it should not have.

Pairing has a second benefit: it forces you to state the boundary. To write the negative counterpart of "an order from 5 days ago should be refunded", you must answer "how many days is inside the window" — a question you can dodge indefinitely writing positive tasks only.

Sit the exam yourself before handing it out

Suppose item seventeen of your twenty-four has a pass rate of zero. Most people conclude the agent is weak and start rewriting the prompt. The likelier explanation is that the task is wrong.

Before the exam paper is printed, the writer works through every question to confirm it is solvable and the key is correct. Evaluation needs the same step, and the instrument is a reference solution: a fully deterministic fake agent with the correct rules written directly into its code.

If the reference solution cannot pass a task, there are two possibilities:

1. The task is wrong — the expected amount is off by a digit, the order id in the prompt is not in the seed, or the criterion demands something the rules make impossible.
2. The grader does not match the task — it expects an escalation, but the grader only inspects refund records.

Either way, that task must not judge any agent until it is fixed. Every score it produces is fiction.

A real case makes the point. A model scored 42 on a public benchmark. Every problem turned out to be on the evaluation side: grading used strict string comparison, so 96.12 and 96.124991 came out unequal; some descriptions were ambiguous; some tasks carried randomness and were not reproducible. After grading was fixed, the same model scored 95 on the same set. Fifty-three percentage points, none of it about model capability.

What a zero and a perfect score are each telling you

The reference solution is the first gate. Once things run, the two extreme scores each carry a meaning.

A pass rate of zero: suspect the task first. Failing a hundred trials out of a hundred intuitively means the agent cannot do this. The more common explanation is a broken task: wrong expectation, a missing order in the seed, a criterion the rules make unreachable. Run the reference solution before you declare the agent incapable.

A perfect pass rate: suspect the discriminating power. A task everybody passes carries no information. The sneakiest variant: the order id the user mentions is not in the seed, so no agent finds it, every agent does nothing, and doing nothing happens to be the expectation. That task scores perfectly forever while telling you nothing.

To catch it, run two degenerate solutions alongside: one that refunds everything, one that does nothing. Three fake agents, one run each, over every task:

• The reference solution fails it, the task is broken.
• All three pass it, the task has no discriminating power. Delete or rewrite it.
• The reference passes and at least one degenerate fails, the task is doing work.

All three fakes are deterministic and need one trial each, so this is cheap, and it removes the two hardest-to-find classes of broken task before you hand the set to anybody.

Keep capability and regression sets apart

One last thing, and it decides both the shape of your files and the number of figures in your report. Yesterday introduced the two ideas; today they land in the file layout.

Mixing both into one suite and reporting a single total is the classic beginner mistake. That total loses both abilities at once: it cannot raise an alarm (two regression failures drown in a dozen capability failures) and it cannot point a direction (the score went up, and you cannot tell whether new capability landed or an old regression got fixed). Tag every task with its set and print two lines. Today's lab shows the gap — regression 100 percent, capability 50 percent, overall 70.8 percent. The 70.8 percent is the only useless one.

A benchmark set is a living thing

A healthy set has a workflow attached: a new failure happens in production, it becomes a task the same day, the paired negative case goes in with it, the reference solution validates both, and they join the regression set. Do that and the same failure never happens twice. A set nobody has touched in six months is measuring the product as it looked six months ago. D7 health-checks the suite itself.

Source Reading

The Anthropic engineering post on evaluation method is the skeleton for the whole day: mining tasks backwards from real failures, twenty to fifty items being enough to start, why two-directional testing is necessary, and the split between capability and regression evaluation. It is also where the 42-to-95 case comes from, and that case is worth reading in full — its value is being a counterexample: everyone's first instinct was that the model was weak, and the entire truth was on the evaluation side. The later passage on not trusting a score unless somebody has read several transcripts is Day 7's closing argument, so skip it for now.

The SWE-bench site offers the other angle: how a widely used public benchmark guarantees its tasks are solvable. Its approach is worth copying — every task carries a real commit, known to turn the tests from red to green, as its reference solution, built backwards from the history of real repositories rather than invented by hand. That also confirms again that questions come from real failures. And notice its several published variants, which reflect something you should expect: once a benchmark is used at scale, broken tasks get discovered in it, and a human-verified subset becomes necessary. The same will happen to your own set.

Hands-On Lab

Today adds two modules on top of D1's kernel: loading and validating the benchmark set, and reference-solution validation.

The set ships as JSON, deliberately: tasks are pure data, not code. A data file can be reviewed by a product manager, shows exactly which item changed in a git diff, and is read straight from CI on Day 6. The cost is that no type checker stands behind it, so a validator is mandatory, and what it has to catch is not syntax errors but the problems that are semantically wrong yet run perfectly well. Two are worth memorizing, because they fail in opposite directions:

// Broken shape one: the expected amount does not match the order amount.
// This task fails forever, and the failure looks like the agent's fault
for (const r of task.expect.refunds ?? []) {
  const order = orders.find((o) => o.id === r.orderId)
  if (!order) {
    issues.push(`expected refund order ${r.orderId} is missing from seed.orders`)
  } else if (order.amountCents !== r.amountCents) {
    issues.push(`expected amount ${r.amountCents} does not match order amount ${order.amountCents}`)
  }
}
 
// Broken shape two: the order named in the prompt is not in the seed at all.
// This task passes forever, because nobody can find the order, nobody does
// anything, and doing nothing happens to be the expectation
const mentioned = task.prompt.match(/[A-Z]\d{4}/)
if (mentioned && !orderIds.has(mentioned[0])) {
  issues.push(`order ${mentioned[0]} named in the prompt is missing from seed.orders`)
}

# Broken shape one: the expected amount does not match the order amount.
# This task fails forever, and the failure looks like the agent's fault
for r in task["expect"].get("refunds", []):
    order = next((o for o in orders if o["id"] == r["orderId"]), None)
    if order is None:
        issues.append(f'expected refund order {r["orderId"]} is missing from seed.orders')
    elif order["amountCents"] != r["amountCents"]:
        issues.append(f'expected amount {r["amountCents"]} does not match order amount {order["amountCents"]}')
 
# Broken shape two: the order named in the prompt is not in the seed at all.
# This task passes forever, because nobody can find the order, nobody does
# anything, and doing nothing happens to be the expectation
mentioned = re.search(r"[A-Z]\d{4}", task["prompt"])
if mentioned and mentioned.group(0) not in order_ids:
    issues.append(f"order {mentioned.group(0)} named in the prompt is missing from seed.orders")

A task that fails forever sends you hunting a bug that does not exist. One that passes forever makes you believe you tested something when you tested nothing. Both are worse than not having the task at all. Once it runs, the output looks like this:

benchmark set 2026-09-14   tasks: 24
 
ok  set validation: format, amounts, polarity, pairing and set labels all consistent
 
ok  reference-solution health check: 24/24 solvable, 0 without discriminating power
 
-- baseline report (seed 20260914, 5 trials each) --
overall      24 tasks  pass@5 100.0%  pass^5  70.8%
capability   14 tasks  pass@5 100.0%  pass^5  50.0%
regression   10 tasks  pass@5 100.0%  pass^5 100.0%

Regression at 100 percent says the things you fixed are still fixed; capability at 50 percent says half the scenarios are not reliable yet. Both convert straight into a next action. The 70.8 percent does neither.

One item on the README's manual checklist is delete all twelve negative tasks and run it again. The score goes up, because the "refund everybody" defect only loses points on negative tasks. That is the most persuasive thirty seconds in the whole day.

Interview Questions

Today's four questions circle the construction and validation of a benchmark set: where tasks come from, where one-sided testing takes a system, what makes a task acceptable, and why the two kinds of evaluation expect different scores. The second is worth sitting with. People who miss it answer "coverage is incomplete". The real answer is that the optimization direction gets bent: the defect does not merely go unpunished, it gets rewarded.

Checklist and Tomorrow

By the end of today you should be able to:

• Name the four real sources of benchmark tasks, and translate one ticket into a complete task
• Use the "two experts agree independently" criterion to find the ambiguous items you wrote
• Write the paired negative task for any positive task, and say what happens if you skip it
• Explain what a reference solution is for, and why it must not read the polarity label
• Say what a zero pass rate and a perfect pass rate should each make you suspect first
• Get all ten assertions green with MOCK=1 pnpm selftest
• See for yourself the run where deleting the negative tasks raises the score

Tomorrow is D3, Let a Model Judge, Then Put the Judge Through an Exam. Every criterion in today's twenty-four tasks lands on the outcome, so each grader is a few lines of code. Real systems always have a portion code cannot judge — was the reply clear, did it answer the question asked. That part needs a model as the judge, and an uncalibrated model judge is itself a system nobody has evaluated. Tomorrow puts the judge through its own exam: position bias, verbosity bias, self-preference bias, and why judging a judge takes an agreement coefficient rather than accuracy.