Managing Context for Tool Results and Retrieval: Loading on Demand, Summarizing and Pruning, Structured Returns
Tool definitions and tool results are the two easiest pieces of context to blow up. Today gives criteria for pruning the tool list, patterns for loading on demand and returning structured results, and a pruner that can quantify the before-and-after difference.
Today's Goals
- Name the two problems that come from a bloated tool list, and give a set of criteria for pruning tools per task
- Implement a tool-result pruner that compresses by a field allowlist, a length cap, and dropping stale results
- Explain which scenarios suit loading on demand versus retrieving up front, and how to combine them into a hybrid strategy
Yesterday treated the piece you write yourself. Today treats the two pieces an external system hands you — tool definitions and tool results — which together took over eighty percent of the D1 profile. Come back when you have read it and tick off the three goals.
Plain-Language Walkthrough
Do not pack the whole wardrobe
You write the system prompt, so its length is yours to set. Tool results are not: call an order lookup once and how many fields come back, and how long each is, is decided by someone else's service. While writing the code you have no idea how big it will be.
That is the root reason it becomes the piece most prone to blowing up. In the D1 lab, one order lookup returned 1,659 tokens and one product search 2,803 — and neither of those endpoints was designed with any thought that a language model would read it while paying per token. They return audit_log, internal_flags, supplier_code, and shelf, because for a web page a few extra fields cost essentially nothing.
So something absurd shows up in your context: the model is paying to read a pile of audit logs it will never use. Worse, that content is not merely wasting budget; it is diluting attention in exactly the way D1 described — the one useful status field, drowned among eighteen state_transition records.
One sentence for the awareness today builds: what a tool returns is written for a program, not for a model. A pruning layer of your own has to sit in between.
And before pruning anything, there is an earlier question to settle: how many tools should you have attached at all? The answer to that changes every number that follows.
Interview questions
An agent has thirty tools mounted and it is clearly struggling. How do you cut the list, and on what basis?Agent 挂了三十个工具,明显吃不消了。你会怎么裁?依据是什么?
Common in ChinaCommon overseasIntermediate#tool-design#tool-budget🔒 Unlock this chapter to see the answer keys.
How do you choose between just-in-time loading and pre-inference retrieval, and how would you combine them?按需加载和预先检索怎么选?混合策略应该怎么搭?
Common in ChinaCommon overseasIntermediate#retrieval#just-in-time#hybrid🔒 Unlock this chapter to see the answer keys.
Why are tool results untrusted input, and does field whitelisting make the concern go away?为什么说工具返回的内容是不可信输入?做了字段白名单裁剪之后还需要担心吗?
Common in ChinaCommon overseasDeep dive#prompt-injection#trust-boundary🔒 Unlock this chapter to see the answer keys.
