Measuring and Tuning: the Token Bill, Context Utilization, Failure-Mode Triage, and a Comprehensive Interview Deep Dive
Fold the previous four days' techniques into one measurable loop: work out the token bill and cache hits, watch two utilization metrics, locate the problem against four failure modes, then work through a comprehensive context-engineering interview deep dive.
Today's Goals
- Work out a task's token bill, and split it into cache writes, cache hits, and new input
- Use the window-occupancy rate and the useful-information ratio to locate which part of the context is the problem
- Match a symptom against four typical failure modes, and give a triage path from symptom to fix
The first four days taught techniques: trimming, pruning, compaction, isolation. Today teaches which technique when — and when to stop. Come back when you have read it and tick off the three goals.
Plain-Language Walkthrough
Weighing the bag on the way home: the bill is not the last weigh-in
Back at the airport you put your bag on the scale, and the number is how heavy the case is at this moment. But if the airline charged by weighing separately on each leg, what you actually pay is the sum of four legs' weights — and you kept adding things along the way, so the later legs are certainly heavier than the first.
A model's bill works exactly that way. Many people quote cost as "how many tokens per request," which is a weigh-in, not a bill. The bill is the running sum of every turn's input across the whole session, because the model has no memory and every turn resends everything before it.
How big is that difference? Work it out on the 20-turn e-commerce support task from today's lab: the stable prefix per turn is 1,612 tokens (system prompt 881 plus tool definitions 731), history grows about 55 per turn, and a tool is called roughly every 3 turns at about 1,424 each time. That adds up to this table:
| Component | Growth pattern | Total over 20 turns |
|---|---|---|
| Stable prefix | Resent verbatim every turn | 32240 |
| Conversation history | Linear growth | 11550 |
| Tool results | Step growth | 89712 |
| Total | 133502 |
The single input on the last turn is 11,256 tokens while the session's bill is 133,502 — nearly 12 times apart. Anyone reporting only the single input understates the cost by an order of magnitude.
That table also answers where to change first: tool results take 67%, so do D3's pruning first; the prefix takes 24%, so D2's trimming comes next; history is only 9%, so D4's compaction comes last. The four days' order is not sorted by difficulty but by this table.
But the table left one thing out, and it turns the conclusion inside out — caching. That is where we start.
Interview questions
How do you compute the token bill for one agent task, and which parts can be cached away?怎么给一个 Agent 算一次任务的 token 账单?哪些部分是可以被缓存掉的?
Common in ChinaCommon overseasDeep dive#token-accounting#prompt-caching🔒 Unlock this chapter to see the answer keys.
When context is the problem, how do you localize which of the four buckets is at fault?上下文出问题的时候,你怎么定位是四块里的哪一块?
Common in ChinaCommon overseasIntermediate#diagnostics#metrics🔒 Unlock this chapter to see the answer keys.
How much context engineering is enough, and how do you know when to stop?上下文工程做到什么程度算够?你怎么知道该停手了?
Common in ChinaCommon overseasIntermediate#tuning#stopping-criteria🔒 Unlock this chapter to see the answer keys.
