Measuring and Tuning: the Token Bill, Context Utilization, Failure-Mode Triage, and a Comprehensive Interview Deep Dive

How caching rewrites the bill

A stable, unchanging prefix can hit the prompt cache. The economics are simple: writing to the cache costs slightly more than the base price, and hitting it costs an order of magnitude less. The common figures are 1.25 times for a write (2 times for a one-hour lifetime) and 0.1 times for a hit.

So 20 turns of prefix cost go from twenty full-price sends to one at 1.25 times plus nineteen at 0.1 times:

• Without caching: 1,612 times 20, or 32,240 effective tokens
• With caching: 1,612 times 1.25, plus 1,612 times 0.1 times 19, or 5,078 effective tokens

84% cheaper, without one line of business code changed. That is also why day two stressed layering by stability with the stable content up front — layering is not only for legibility, it directly determines how long the cached prefix can be.

But caching has a threshold: the prefix must reach the minimum cacheable length, or it simply does not apply, and no error is raised. That length depends on the model; Claude Sonnet 5, for example, is 1,024 tokens, while Claude Opus 5 and Fable 5.1 are 512.

Now add D2's and D3's optimizations and see what happens: the system prompt goes from 881 to 425, the tools are pruned from 8 to 3 so definitions go from 731 to 308, and the total prefix goes from 1,612 to 733.

733 is below 1,024. The cache stops applying. The prefix's effective cost rises from 5,078 to 14,660 — nearly three times more expensive after the optimization.

With all three optimizations counted, the total bill goes from 133,502 to 21,620, a saving of 83.8%; with caching factored in, from 106,340 to 21,620, a saving of 79.7%. Report both numbers; reporting only the first is flattery.

Two utilization metrics

The bill tells you how much was spent, and metrics tell you whether it was spent well. Two are enough, and more go unread.

The window-occupancy rate = a single turn's input tokens divided by the window limit. It governs whether things will burst, and it is the leading indicator of context rot. The rule of thumb: act above 50%, rather than waiting to approach the limit.

The useful-information ratio = the tokens later steps genuinely use divided by total tokens. It governs whether the spending was worth it. How to measure it? A workable approximation is to run D3's pruner over the context, and what survives the prune is the numerator.

These two often move out of step, and the moments they do are precisely the dangerous ones. Take the scenario from today's lab:

Metric	Before	After	Threshold
Window-occupancy rate	5.6%	0.7%	Act above 50%
Useful-information ratio	19.1%	100%	Below 20% means you are hauling garbage

The occupancy rate is only 5.6%, nowhere near bursting, and looks entirely fine; but the useful-information ratio is only 19.1%, meaning eighty percent of the tokens are generating interference for the model while being billed at full price. Anyone watching only occupancy will wait until the day something breaks before optimizing.

Four failure modes

With metrics in hand, triage no longer relies on guessing. Context problems present in basically only four ways:

Mode	Typical symptom	Metric that fires	First technique
Too full	Misses early instructions, violating constraints written plainly in the system prompt	High window-occupancy rate	Compact the history, clear tool results
Cannot find it	The information is in the window and the model says it does not have it	Low useful-information ratio	Prune tool results, return structured data
Cannot say why	Answers to the same class of question are unstable, varying between runs	Occupancy not high but error rate is	Fix the system prompt's altitude, merge overlapping tools
Drifting off	Fine for twenty turns, then starts violating the original constraints	❌ appears in the retention check around compaction	Increase retained turns, put hard constraints in the must-keep section of the summary

The most misdiagnosed of the four is "cannot find it," because its symptom looks so much like insufficient model capability — you start thinking about a stronger model when the real problem is that the line was buried among eighteen audit log records. The criterion is: print the context and search it by hand for that line. If it is there, it is a context problem, not a model problem.

The second most misdiagnosed is "cannot say why." It usually gets blamed on model instability, but if answers to one class of question waver between two runs, it is most likely two rules fighting inside the system prompt, or two tools with overlapping responsibilities making the model flip-flop at a decision point. Searching for contradictory rules is far more useful than switching models.

The tuning loop: change one thing at a time

With the bill, the metrics, and the mode table, what remains is a very plain loop:

1. Fix a set of cases. Not many — 10 to 20 covering the normal, the edge, and the longest path is enough — but they must be fixed, since swapping cases midway voids every earlier measurement.
2. Measure the baseline: run once and record the bill, both metrics, and the count of each failure class.
3. Pick one change from the mode table, and change one thing at a time.
4. Re-measure with the same cases, recording both the expected change and the actual change — especially the ones that disagree.
5. When they disagree, go back and work out why. That step is where the learning is, and skipping it degrades this loop into blind tuning.

The actual-change column of step four is where the whole table's value sits. In today's reference answer, one of three changes came out opposite to expectation: pruning the tools from 8 to 3 did lower the token count, but the prefix fell below the minimum cacheable length and the bill rose. Had we watched only the token count, that step would have been recorded as an unqualified win.

Last comes the stopping criterion. Context engineering is work that can go on forever, so decide in advance when to stop:

1. The useful-information ratio holds above 50%, with no room to rise across three consecutive measurements.
2. The single-turn window-occupancy rate stays under 50% on the longest case.
3. The most recent change brought the bill down by under 5%.

The third is the real brake: a saving under 5% means what is left is necessary overhead, and squeezing further trades accuracy for money.

The comprehensive interview deep dive: how to answer context engineering questions

Finally, the interview angle. Context engineering questions almost all probe the same thing — whether you treat it as a bag of tricks or as a measurable engineering problem. Three general principles for answering:

First, always give the breakdown before the conclusion. Asked what to do about a context that is too long, answering "compact it" is the worst answer, because it skips looking at which piece is long. The right opening lays out the four pieces, explains that they grow differently and must be treated separately, and only then says which one you would touch first and why.

Second, pair every technique with its cost. Pruning's cost is possibly cutting a field needed later, so you leave a retrieval identifier; compaction's cost is losing things without a marker, so you have a retention check; subagent isolation's cost is total tokens rising to a dozen-odd times chat. Give benefits without costs and you will be pressed on it, and when you are, you will be on the back foot.

Third, give numbers. "Tool results are usually the bulk" is weaker than "I measured an e-commerce support session where tool results were 72.9%, and I had guessed conversation history beforehand." A specific number plus the sentence "and I guessed wrong" is the most persuasive combination in this kind of question — it proves both that you actually measured and that you know intuition is unreliable here.

If the interviewer keeps digging, it usually lands in three directions: the relationship between caching and the bill (today's second section), how to verify compaction (D4), and when to reach for multi-agent (D4, where the answer is usually "most of the time you should not"). All three have specific numbers you can cite, so when preparing, memorize the numbers rather than the conclusions.

Source Reading

Hands-On Lab

The fifth criterion is not a trick. Three changes all matching expectation usually means you did not actually re-measure but copied the expectation into the actual column. This report's entire value is that it accepts only numbers; the moment you start filling in impressions, it degrades into a status update.

1. Run the D1, D3, and D4 labs each once and keep the outputs at hand.
2. Open the starter's report template and fill it from section 0 downward, with every placeholder traceable to an actual run.
3. Do not skip the caching column in section 2: look up the minimum cacheable length of the model you use first, then compute the effective cost.
4. Pick one of the four modes in section 4, and write the basis as "because such-and-such metric is such-and-such" rather than as an impression.
5. Open the solution and compare, focusing on the metric-improved-bill-worsened example in its fifth change record.

Interview Questions

Today's 3 questions are in the question bank below, weighted toward the relationship between the token bill and caching, the definition of the context utilization metrics, and the triage path for failure modes. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Work out a task's token bill, and split it into cache writes, cache hits, and new input
• Use the window-occupancy rate and the useful-information ratio to locate which part of the context is the problem
• Match a symptom against four typical failure modes, and give a triage path from symptom to fix
• Explain how trimming a system prompt can make the bill more expensive, and name three fixes
• All 5 acceptance criteria of the lab pass, including the one requiring at least one change to defy expectation
• Answer at least 2 of the 3 interview questions without looking at the key points
• Look back over the five days' artifacts: the profiler script, the trimmed instruction file, the pruner, the compactor, and the tuning report — together they are a presentable context engineering portfolio

This is the course's last day, so there is no preview of tomorrow, only a reminder: these five days taught trade-offs, not configuration. The numbers — thresholds, retained turns, allowlisted fields — all have to be re-measured on a different project; what you take away is the practice of measuring first, ordering by share, changing one thing at a time, and pairing every technique with a verification.

To keep expanding outward, two directions live in the same batch of courses: where tools come from and how to wire them cleanly is day 1 of MCP in 7 Days; how to package working experience into on-demand capability bundles is day 1 of Agent Skills in 7 Days. One sentence for the three courses' division of labor: MCP handles the wiring, Skills handle the experience, and context engineering handles the trade-offs.