Evaluation and Cost: Designing a Benchmark Set, Computing Pass Rate, and Reading Token Usage and Cache Hits
Tuning an agent by gut feel only makes it worse: design a benchmark set of tasks that can be scored automatically, get out a pass rate, average turn count, and token usage, compute the cache hit rate, then use a before-and-after comparison of one prompt change to show why evaluation must come before optimization.
Today's Goals
- Design an automatically-scorable agent task benchmark set, and explain what the scoring function should look at
- Compute four metrics: pass rate, average turn count, token usage, and cache hit rate
- Show the full evaluation-drives-optimization flow with one before-and-after comparison
Nineteen days of adding things to mca; today is the first time we turn around and measure whether it is any good. When you finish, scroll back up and tick the three goals off.
Plain-Language Walkthrough
The probation review: measure with a test set
Three months in, does the new hire go permanent?
The cheap way is to ask whoever trained them, "how is this person?" The answer is almost always "pretty good" — which sounds like information and is not, being decided by the most recent incident and the speaker's mood today. Serious companies hand over a fixed set of tasks instead and score the result against one ruler. The value of that set is not that it measures how clever the person is. It is that next quarter it is still the same set, so "better than last quarter" acquires a meaning for the first time.
Tuning an Agent is the same situation. You change one sentence in the system prompt, try it three times in a terminal, and decide it "feels smoother." Far too much went into those three runs: the model took a good path, you phrased the request more clearly, the sandbox repo still carried last run's edits. People who tune an Agent by feel make it worse over time, because every round they are treating noise as signal.
So today's build is deliberately plain: five tasks, each scorable by a piece of code, run in one shot, printing four numbers. You have seen task one many times — run the tests, fix the failing case, run them again to prove it is green — the baseline task since day four.
Interview questions
How do you design a benchmark set for a coding agent, and should the grader look at the end state or at the process?怎么给一个 Coding Agent 设计基准集?判分函数该看终态还是过程?
Common in ChinaCommon overseasBasic#benchmark-design#grading🔒 Unlock this chapter to see the answer keys.
Which metrics would you use to judge a coding agent, and why is pass rate not enough?你会用哪几个指标衡量一个 Coding Agent 的好坏?为什么通过率不够?
Common in ChinaCommon overseasIntermediate#metrics#cost#cache🔒 Unlock this chapter to see the answer keys.
Results for the same task vary between runs. How do you tell whether a change actually helped, given that noise?同一道题多次运行结果不稳定,你怎么在这种噪声下判断一次改动是不是真的有效?
Common in ChinaCommon overseasDeep dive#variance#ab-testing#evaluation🔒 Unlock this chapter to see the answer keys.
