Dayward AIWeChat Mini Program
Build Your Own Coding Agent in 21 Days
Week 3 · D20🔒 LockedAbout 4 hours

Evaluation and Cost: Designing a Benchmark Set, Computing Pass Rate, and Reading Token Usage and Cache Hits

Tuning an agent by gut feel only makes it worse: design a benchmark set of tasks that can be scored automatically, get out a pass rate, average turn count, and token usage, compute the cache hit rate, then use a before-and-after comparison of one prompt change to show why evaluation must come before optimization.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Design an automatically-scorable agent task benchmark set, and explain what the scoring function should look at
  2. Compute four metrics: pass rate, average turn count, token usage, and cache hit rate
  3. Show the full evaluation-drives-optimization flow with one before-and-after comparison

Nineteen days of adding things to mca; today is the first time we turn around and measure whether it is any good. When you finish, scroll back up and tick the three goals off.

Plain-Language Walkthrough

The probation review: measure with a test set

Three months in, does the new hire go permanent?

The cheap way is to ask whoever trained them, "how is this person?" The answer is almost always "pretty good" — which sounds like information and is not, being decided by the most recent incident and the speaker's mood today. Serious companies hand over a fixed set of tasks instead and score the result against one ruler. The value of that set is not that it measures how clever the person is. It is that next quarter it is still the same set, so "better than last quarter" acquires a meaning for the first time.

Tuning an Agent is the same situation. You change one sentence in the system prompt, try it three times in a terminal, and decide it "feels smoother." Far too much went into those three runs: the model took a good path, you phrased the request more clearly, the sandbox repo still carried last run's edits. People who tune an Agent by feel make it worse over time, because every round they are treating noise as signal.

So today's build is deliberately plain: five tasks, each scorable by a piece of code, run in one shot, printing four numbers. You have seen task one many times — run the tests, fix the failing case, run them again to prove it is green — the baseline task since day four.

Sign in and link WeChat to read

Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.

Mini program QR code

Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.

Interview questions

  • How do you design a benchmark set for a coding agent, and should the grader look at the end state or at the process?怎么给一个 Coding Agent 设计基准集?判分函数该看终态还是过程?
    Common in ChinaCommon overseasBasic#benchmark-design#grading

    🔒 Unlock this chapter to see the answer keys.

  • Which metrics would you use to judge a coding agent, and why is pass rate not enough?你会用哪几个指标衡量一个 Coding Agent 的好坏?为什么通过率不够?
    Common in ChinaCommon overseasIntermediate#metrics#cost#cache

    🔒 Unlock this chapter to see the answer keys.

  • Results for the same task vary between runs. How do you tell whether a change actually helped, given that noise?同一道题多次运行结果不稳定,你怎么在这种噪声下判断一次改动是不是真的有效?
    Common in ChinaCommon overseasDeep dive#variance#ab-testing#evaluation

    🔒 Unlock this chapter to see the answer keys.

Comments