Concept Questions, Part 2: The LLM Fundamentals That Change an Agent Engineer's Decisions
Interviews always ask about how LLMs work, but an agent engineer does not need to derive formulas; you need to know which principles change your choices. Today covers only five: why the KV cache means your prefix must stay stable, why tool calls want a low temperature, how to choose between retrieval, fine-tuning, and a long context, how the sycophancy that preference optimization brings affects a judge model, and how batching, speculative decoding, quantization, and mixture of experts change time to first token and cost. Each one ends on what an agent engineer should therefore do. The interviewer agent learns to score: trade-off and evidence points count double, and every quote must really be in the answer.
Today's Goals
- Can explain how the KV cache relates to prompt caching, and point out which ways of writing a system prompt break the cache
- Can choose between retrieval, fine-tuning, and a long context and state the cost of each, and can say what the sycophancy from preference optimization means for designing a judge model
- Can implement a scorer that weights answer points by type, and use quote verification to throw out evidence the model made up
Yesterday's frameworks all sit on top of the model. Today goes one layer down. When you finish, come back to the top and check off the three goals.
Plain-Language Walkthrough
You can drive without building an engine
Nobody needs to know how to build an engine to learn to drive, but a few engine principles really matter: do not floor it on a cold engine, use a low gear on a long downhill, fuel use depends on revs. What they have in common is that they change how you drive. How pistons are cast or how a crankshaft is balanced is nice to know, and irrelevant to driving.
LLM fundamentals work the same way for an agent engineer. Interviews will ask about Transformers, attention, the KV cache, and RLHF, and question banks online take them down to the formulas. But your time is limited, and the principles to learn first are the ones that make you choose differently:
- The KV cache: how to write your system prompt
- Sampling temperature: what to set for tool calls
- Retrieval, fine-tuning, long context: where knowledge should live
- Preference optimization: whether and how to use a model as a judge
- Inference serving: why the first token is slow and where the money goes
Today covers only these five, and each ends on what an agent engineer should therefore do. Answer in the same order in an interview: one sentence on the principle, then one on which decision it changed for you. When the interviewer hears the second half, they know you are not reciting.
Attention itself is not taught from scratch here. One sentence is enough: to generate each new token, the model looks back at every earlier token, by comparing the current position's query vector with every earlier position's key vector and taking a similarity-weighted mix of their value vectors. The KV cache grows directly out of that sentence.
Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.
Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.
Interview questions
What is a KV cache, and how does it relate to the cost of running an agent?KV Cache 是什么？它和 Agent 的成本有什么关系？
Common in ChinaCommon overseasDeep dive#kv-cache#prompt-caching#cost🔒 Unlock this chapter to see the answer keys.
RAG, fine-tuning, or a long context window: how do you choose?RAG、微调和长上下文，你怎么选？
Common in ChinaCommon overseasIntermediate#rag#fine-tuning#long-context🔒 Unlock this chapter to see the answer keys.
Why is temperature usually set low for tool calls?为什么工具调用时温度一般设得很低？
Common in ChinaCommon overseasBasic#sampling#temperature#tool-calling🔒 Unlock this chapter to see the answer keys.
How do preference-optimization methods like RLHF and DPO affect using an LLM as a judge?RLHF、DPO 这类偏好优化，对你用大模型当裁判有什么影响？
Common in ChinaCommon overseasDeep dive#rlhf#llm-as-judge#evaluation🔒 Unlock this chapter to see the answer keys.