Dayward AI
中文
Week 1 · D1About 6 hours

LLM API Basics: messages/roles, Tokens, Streaming, Temperature; What an Agent Actually Is

Starting from messages/roles, tokens, streaming, and temperature, get clear on what an LLM API actually does, and what an agent has that a plain chat doesn't.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Explain in your own words the relationship between tokens, the context window, and the messages array
  2. Write a TypeScript CLI that prints a streamed reply, and switch between different models
  3. State that Agent = model + loop + tools + memory, and give one production example

When you have read today's material and finished the lab, come back to the top of this page and tick off those three goals one by one — this is the first real check-in of your next thirty days.

Plain-Language Walkthrough

A large language model is a giant autocomplete engine

Start with something you use every day: the suggestion bar on your phone keyboard. You type "see you" and it offers "tomorrow", "soon", "later" above the keys. It does not understand your plans and it does not care whether you actually show up. It has simply counted, across an enormous pile of text, what tends to follow those two words. A large language model (LLM) is doing the same job, just with a much longer memory of what it has read and much better aim: hand it a stretch of text and it guesses the single most plausible next chunk, appends that guess, guesses again, and keeps going until it decides to stop. Every "conversation" you have ever had with a chat product is, underneath, the whole history flattened into one long stretch of text and handed to the model to continue — and the continuation is the reply you see on screen. Once that clicks, every strange behavior you run into later starts to make sense: the model is not deliberating over your question, it is computing the most plausible continuation of the text in front of it.

So does the model see characters, or something else? It sees tokens. A token is the smallest unit the model handles internally: sometimes a whole short word, more often a word fragment, a piece of punctuation, or a common suffix. In English one word averages roughly 1.3 tokens; a word like "unbelievable" may split into three. That number looks like trivia, but it decides two very practical things: how much the model can look at in one shot (next section), and how much you pay — essentially every LLM API bills by the token, counting both the history you send in and the reply that comes back, so a long conversation costs more per turn than a short one. Fix that exchange rate in your head now and no pricing page will ever confuse you again. It matters doubly for agents: an agent calls the model repeatedly inside a loop and stuffs every tool result back into the history as it goes, so its token consumption grows far faster than a plain chat's. You cannot control agent cost without understanding tokens.

How does a model "continue" text?1/4
Weather

Candidates for the next token

(scoring hasn't started yet)

The model receives a piece of text and is about to continue it. It doesn't understand "Weather" — it's just computing the most likely next token.

What the API for talking to a model looks like

Now that you know the model is continuing text, look at how you actually call it. Strip away every SDK and framework and one raw conversation request is a single HTTP POST that looks like this:

chat.js
const res = await fetch('https://openrouter.ai/api/v1/chat/completions', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'openai/gpt-4o-mini',
    messages: [
      { role: 'system', content: 'You are a patient programming tutor.' },
      { role: 'user', content: 'Explain what a token is in one sentence.' },
    ],
  }),
})
const json = await res.json()
console.log(json.choices[0].message.content)

Walk the fields one at a time. model picks which model to use; the value here goes through OpenRouter, an aggregation gateway where one key buys you the same interface across dozens of vendors — OpenAI, Anthropic, Google, open-weight models — so switching models means editing one string rather than integrating another SDK. That is why every lab in this course defaults to it. messages is an array holding the whole conversation, and each entry carries a role: system sets the model's identity, boundaries, and output style and normally sits first; user is what you or your user said; assistant is what the model said earlier. Interleave the three roles, flatten the array, and you have the stretch of text the model is asked to continue. The response comes back as JSON with the actual reply at message.content inside the first element of choices. Memorize that shape — every agent you write over the next thirty days ultimately emits a request of exactly this form, with the messages array simply growing longer as tool results pile into it.

The context window: how big is the model's desk

Think about the desk you work at. Its surface is a fixed size, so once the paper piles up, something has to go into a drawer or into the bin or the new page simply will not fit. A model's context window is that desk: a hard ceiling on the total tokens one request may hold, input plus output together, ranging from tens of thousands to over a million depending on the model. Cross it and the request errors outright, or something upstream quietly truncates the oldest material and the model never sees it at all.

Here is the fact newcomers overlook most often: the model has no memory. Every HTTP request is brand new and stateless; it does not "remember" your previous sentence the way a person would. What we call a multi-turn conversation is entirely a messages array that you maintain on your side: append each user message as it arrives, and append the model's reply too, so that both travel along as history on the next request. Forget to store a reply back into the array and the model develops amnesia on the very next turn and answers the wrong question. This also explains why long conversations get expensive: each additional turn lengthens the history, and the entire history is re-sent and re-billed alongside every new question, so cost grows roughly linearly in the number of turns. D6 tackles what to do once this desk is full — summarization, sliding windows, the context-engineering toolkit — and D12 moves genuine long-term memory off the desk entirely into a searchable external store you draw from on demand instead of hauling everything back onto the surface each time. For today, one sentence is enough: the model has no memory, and carrying the history is your job.

How the context window gets filled up1/5
Used 20 / 100 tokens
system persona20 tok
The context window is the model's desk — a fixed size. The system persona goes on first, and it usually has to stay there the whole time.

Streaming output: the typewriter effect is not an animation

You have surely noticed that AI chat products reveal their replies a few characters at a time, like an old typewriter striking the page. That is not a flourish a design team added; it is how the server genuinely works. The model emits one token at a time — back to the autocomplete picture from the first section — so if the server insisted on waiting for the whole paragraph before sending anything, the user would sit in front of a blank screen for another ten or fifteen seconds. Push each token out the instant it exists and the first character lands inside a second. The gap in experience is enormous. The protocol for generating and pushing at the same time is usually SSE (Server-Sent Events), a one-way text protocol layered on ordinary HTTP. Set stream to true in the request body and the server stops returning one complete JSON document and starts returning a run of text blocks like these:

TextText
data: {"choices":[{"delta":{"content":"To"}}]}
 
data: {"choices":[{"delta":{"content":"ken"}}]}
 
data: [DONE]

Each block opens with data: and carries a small JSON payload whose delta.content is the newly generated fragment. When everything is finished you receive a line reading data: [DONE]. Turning that run of blocks back into a stream of characters on screen takes about twenty lines:

stream.js
const reader = res.body.getReader()
const decoder = new TextDecoder()
let buffer = ''
while (true) {
  const { done, value } = await reader.read()
  if (done) break
  buffer += decoder.decode(value, { stream: true })
  const lines = buffer.split('\n')
  buffer = lines.pop() ?? '' // the tail may be half a line, keep it for the next round
  for (const line of lines) {
    if (!line.startsWith('data: ')) continue
    const data = line.slice('data: '.length).trim()
    if (data === '[DONE]') return
    const delta = JSON.parse(data).choices?.[0]?.delta?.content
    if (delta) process.stdout.write(delta)
  }
}

Read raw bytes a chunk at a time with getReader, turn them into text with TextDecoder, then split on newlines and handle one line at a time. This is unlike how you normally consume an API response, because the network will not politely package the data one event per delivery. And this read-a-chunk, parse-by-line pattern is not only for typewriter printing: when an agent pushes its intermediate state to a frontend as it works — what it is currently thinking about, which tool it is calling — it rides the same machinery, so the user watches progress instead of staring at a spinner in front of a black box.

A network packet can cut an SSE line in half1/5

What the server wants to send (the full message)

data: {"choices":[{"delta":{"content":"Hi"}}]}
data: {"choices":[{"delta":{"content":" there"}}]}
data: [DONE]

The network packets that actually arrive

(nothing received yet)

The half-line left in the buffer

(empty)

Complete events parsed so far

(none yet)
The server sends text in SSE format: one event per line, starting with data:, separated by blank lines. In an ideal world you'd just read line by line.

Stream resumption: three kinds of disconnect are three different problems

With streaming explained, the next question is always: what happens when the network drops? It looks like one question and it is three, and the answers do not transfer between them. Conflating them is a common way to lose points in an interview.

First, the browser's native EventSource. This is the SSE client the specification defines, and it reconnects by itself: the server numbers events with an id: field, the browser remembers the last number it received and sends it back in a Last-Event-ID header on reconnect, so the server knows where to resume. A retry: field even sets the reconnect interval. None of that costs you a line of code. But it carries two hard limits: it can only issue GET requests, and the response content type must be text/event-stream. A status code other than 200, or the wrong content type, and the connection is declared failed with no retry. In the other direction, when the server wants to stop the client from reconnecting at all, the standard move is to answer 204.

Second, the kind you will actually meet. An LLM endpoint has to be POSTed to, because messages lives in the request body, and EventSource can only issue GET. So real code is fetch plus a hand-written parser, exactly like the snippet above. That means none of the automatic reconnection from the first case applies — when it drops, it is dropped, and the retry logic is yours to write. The way to recover is continuation rather than restart: send the fragment you already received back as context and let the model carry on from there, saving both the time and the money of regenerating it. One boundary is worth remembering: structured content such as a tool call or a reasoning block cannot be half-recovered, so you can only resume from the most recent complete text block.

Third, the user closed the tab. No amount of frontend retry logic helps here, because the frontend no longer exists. The only way out is to let the generation live independently of that client: while the server pushes tokens to the current connection, it also writes the same content somewhere like Redis and records that stream's id against the session. When the user comes back, the frontend hands the session id to a dedicated resume endpoint, the server finds that stream by id and continues pushing, and answers 204 when there is nothing to resume. The price is one more store to run, an expiry sweep to maintain, and the concurrency question of one stream being consumed by several connections at once. That is precisely the message bus and state machine you will build by hand in mini-koda, the W2 milestone project.

The three layers of reconnection: not the same problem1/5
Layer 1 · Native browser EventSourceGET only
ConnectedConnection established; the server timestamps every event with id:
Layer 2 · LLM API (fetch + POST)EventSource can't help here
ConnectedA POST goes out with messages; the response body is SSE-formatted
Layer 3 · The user closes the tab and comes backThe client is already gone
ConnectedThe server writes the stream into Redis as it generates, and records an activeStreamId
The three kinds of "disconnect" are actually three completely different problems. First, the normal state: all three paths are running along fine.

Temperature and sampling: when 0, when 0.7

Remember the autocomplete picture. At each step the model is really scoring the candidates for what comes next, producing a probability distribution, then drawing one candidate from it. temperature controls how freely that draw happens — picture it as how heavily the dice are loaded. Set temperature to 0 and the model always takes the top-scoring candidate, never rolling at all, so the same input yields essentially the same output every time. Raise it and the model becomes willing to gamble, occasionally picking a lower-ranked but still sensible candidate, which makes the output more varied and less predictable. top_p is a different knob for the same property: instead of reshaping the distribution, it draws only from the smallest set of candidates whose probabilities add up to p, which amounts to discarding the obviously implausible options before the draw. The two appear side by side in every set of docs, but in practice you tune one or the other — moving both makes it hard to tell which one caused what.

Which value you want depends on the job:

SituationSuggested value
Structured output, tool-call arguments, classification0 (or very near it)
Everyday questions, writing docs, writing code0.2 to 0.5
Creative writing, brainstorming, casual conversation0.7 to 1.0

That table is unusually practical for agent work, because so many steps inside an agent — deciding whether a tool is needed at all, filling in that tool's arguments — are really classification or extraction. Those steps want stable, reproducible, testable results, so they almost always run at temperature 0. Only the final natural-language reply the user reads is a candidate for turning the dial up so the tone is less wooden.

From chat to Agent: what is the difference

We have now taken the whole act of talking to a model apart: assemble a messages array, send it, the model continues the text, a streaming protocol pushes the continuation back character by character, and temperature decides how obediently it continues. That is the entirety of a chatbot — one question, one answer, then it is over, and the model has no power to affect anything outside the conversation.

So what does an agent add? One formula: Agent = model + loop + tools + memory. The loop means the agent stops answering once and clocking off, and instead walks a small circuit repeatedly on its own: work out what to do (think), carry out an action (call a tool), look at how that turned out (observe), and take the result into deciding the next step, until the goal is reached. Tools mean the model inside that loop no longer only emits text; it can request that a real action be performed — query a database, send a message, run a calculation — and therefore change the world outside instead of only producing prose for a human to read. Memory means the loop is not confined to one conversation: across turns and across separate sessions, the agent has to hold on to what matters instead of starting from nothing every time. None of those three comes with the chat API you learned today; an engineer has to build each one around the model. That is exactly what the coming weeks add, one piece at a time — and tomorrow we hand-write that think, act, observe loop with no framework at all and see how few lines it really is.

Worth one warning in advance: the moment an agent like that serves thousands of users in production, a single loop is nowhere near enough. Real systems tend to split accepting user requests from actually running the agent loop into two layers with a message bus between them, so one user's slow task cannot hold everyone else up. Each task's progress is recorded as a state machine — queued, running, waiting on a tool, done, failed — to make it traceable and recoverable. And every model call's token count and cost needs a metering component tallying it continuously, or you will burn through a budget without noticing. Splitting the gateway from the worker, the message bus, the state machine, the cost ledger: that infrastructure is what you will build by hand in mini-koda, the W2 milestone project. Today it is enough to know it exists and why it is needed.

The agent loop: think → call tool → observe1/5
Think
Call tool
Observe
↺ Back to thinking, until the goal is met

The messages array (the whole thing gets resent every round)

userWhat's the temperature in Beijing today?
The user asks a question. So far, it's no different from an ordinary chat.

Source Reading

Hands-On Lab

🧪 D1 lab: a streaming CLI chat program

Code location: labs/agent-30days/day-01-streaming-cli

Acceptance criteria:

  1. You type a sentence and the reply prints progressively, not all at once after a long wait.
  2. Ask two questions in a row and the model remembers the first one in the second reply, proving the history was carried along.
  3. Changing MODEL still produces a working reply, and you can observe a difference in style or speed between models.
  4. Each turn prints the prompt and completion token counts from usage.

Before you write anything, try the panel below against your own key: fill in the address and the secret, hit the connection test, and only start coding once it goes through. It saves you most of the "is my code wrong or is my key wrong" debugging. The secret stays in your own browser — this site's server neither receives nor records it. Hit the simulated network drop midway through a generation and you will see the half-finished reply from the previous section with your own eyes.

LLM Connection Lab

Fill in your own endpoint address and API key, and test the connection directly from your browser.Your API key stays in your browser only—our server neither receives, logs, nor forwards it—this page involves no server-side processing, requests go directly from your browser to the target service.

Follow the five steps below against starter/ in labs/agent-30days/day-01-streaming-cli, and open solution/ only when you are stuck:

  1. Start with a single non-streaming request: leave stream off, fetch the complete JSON response, and print choices[0].message.content to confirm the key and the network both work.
  2. Set stream to true in the request body, read the response chunk by chunk with getReader, parse the lines beginning with data: as SSE events, and print each delta until the typewriter effect appears.
  3. Wrap a multi-turn loop around it with node:readline/promises: on each turn append both the user input and the model's reply to the messages array, and verify the model remembers the previous turn.
  4. Add a MODEL environment variable, run it again against a different model id, and compare the two replies for style and speed.
  5. Print the usage field from the response and see how many tokens that whole conversation actually cost.

Interview Questions

Today's ten questions are in the bank below, the last five of them dedicated to disconnects and resumption. Expand a question and read the analysis before the key points — practicing the derivation beats memorizing the answer. Each one is tagged for the China-domestic or overseas market so you can prioritize by where you are applying.

Checklist and Tomorrow

  • Explain in your own words the relationship between tokens, the context window, and the messages array
  • Write a TypeScript CLI that prints a streamed reply, and switch between different models
  • State that Agent = model + loop + tools + memory, and give one production example
  • Say clearly what the three layers of stream resumption are, and why the first layer's automatic reconnection is unavailable on a POST stream
  • All 4 acceptance criteria of the lab pass
  • Answer at least 6 of the 10 interview questions without looking at the key points

Tomorrow (D2) we drop every framework and hand-write an agent loop that can call tools, verifying in code what today's "loop plus tools" actually looks like. Hand-writing first and learning the framework second is a deliberate order: a framework wraps this loop into a couple of API calls, but if you have never implemented it once yourself, then the first time the framework throws an error or behaves unlike your expectation you will have no idea which layer to look at. Understand the skeleton first, enjoy the convenience second, and you will not end up as an engineer who can only fill in configuration.

Interview questions

  • What are tokens and the context window, and how do they shape agent design?什么是 token 和上下文窗口？它们如何影响 Agent 的设计？
    Common in ChinaCommon overseasBasic#llm-basics#context

    How to reason about it · think before answering

    1. First decide whether this asks for definitions or engineering consequences; a definition-only answer reads as inexperienced.
    2. Follow the causal chain: tokens are the unit of billing and length, the window caps that unit, models are stateless so history is resent every turn, cost grows with turns, hence context engineering.
    3. The differentiator is why agents suffer more: a loop calls the model repeatedly and appends tool results back into history.
    4. Close with concrete tactics: sliding window, summarization, externalized long-term memory, and the cost of each.
    5. Expect the follow-up: why compress before the window is full? Long contexts dilute attention and raise latency and cost.

    分析过程 · 先想清楚再作答

    1. 先判断这题问的是「概念」还是「工程后果」。只答定义会被认为没做过工程，必须落到设计影响上。
    2. 从一条因果链推：token 是计费与长度的计量单位 → 窗口是这个单位的上限 → 模型无状态、历史每轮重发 → 成本随轮数增长 → 所以必须做上下文工程。
    3. 关键要点出在「Agent 比聊天更严重」：Agent 在循环里反复调模型，还要把工具返回结果也塞回历史，增长速度快得多。
    4. 结论给出具体手段：滑动窗口、摘要压缩、长期记忆外置到检索系统，并说明各自代价。
    5. 可以预期的追问：窗口没满为什么也要压缩？答案是长上下文会稀释注意力、抬高延迟与成本，不是塞满了才处理。

    Key points

    • A token is the smallest unit the model processes; roughly 1.3 tokens per English word
    • The context window caps input + output tokens per request; beyond it you truncate or compress
    • Models are stateless, so the full history is re-sent every turn and cost grows with length
    • Hence context engineering: sliding windows, summarization, and external long-term memory

    答题要点

    • token 是模型处理文本的最小单位，大致 1 个汉字 ≈ 1–2 token，1 个英文单词 ≈ 1.3 token
    • 上下文窗口是一次请求里输入 + 输出 token 的上限；超出就要截断或压缩
    • 模型没有记忆，历史必须每轮重新塞进 messages，所以长对话的成本随轮数线性增长
    • Agent 设计因此要做上下文工程：滑动窗口、摘要压缩、把长期记忆外置到检索系统
  • What do the system / user / assistant roles do, and why does system exist?messages 里的 system / user / assistant 三种角色各起什么作用？为什么要有 system？
    Common in ChinaCommon overseasBasic#llm-basics#prompt

    How to reason about it · think before answering

    1. The discriminating half is 'why does system exist'; the first half is a warm-up.
    2. Explain that the three roles are structural markers over one continuous text the model continues.
    3. Then the why: rules placed in user are just another turn and get diluted over dozens of turns; system keeps stable weight and can be governed centrally.
    4. Add production nuance: a real system prompt is templated — persona plus tool docs plus memory plus runtime facts.
    5. Likely follow-up: can system go last? Possible but unwise — models weight earlier instructions more and it breaks prompt-cache prefixes.

    分析过程 · 先想清楚再作答

    1. 题眼在后半句「为什么要有 system」——前半句是送分，后半句才是区分度所在。
    2. 先说清三者构成一段可被模型续写的完整文本，角色是给这段文本打的结构化标记。
    3. 再回答「为什么」：如果把规则写进 user，它就只是对话里的一句话，会被后续几十轮对话稀释；放进 system 才能保持稳定权重，且便于产品侧统一管控、单独灰度。
    4. 补一条生产视角：真实的 system prompt 通常是模板拼出来的——人设 + 工具说明 + 记忆片段 + 当前时间，而不是一个写死的字符串。
    5. 常见追问：能不能把 system 放在最后？可以但不推荐，多数模型对靠前的指令更敏感，且会破坏缓存前缀。

    Key points

    • system sets identity, constraints and output format; it sits first and carries more weight
    • user is the human turn, assistant is the model's prior replies; they alternate
    • Rules live in system so they are not diluted by later turns and can be controlled centrally
    • In production the system prompt is templated: persona + tool docs + memory + runtime facts

    答题要点

    • system 设定身份、边界与输出格式，通常放在最前面，权重高于普通对话
    • user 是用户输入，assistant 是模型历史回复，两者交替构成对话记录
    • 把规则放 system 而不是 user，是为了让规则不被后续对话冲淡，也便于产品统一管控
    • 生产里 system prompt 往往由模板拼接：人设 + 工具说明 + 记忆 + 当前时间等动态信息
  • Why do LLM apps stream responses, and how do you choose between SSE and WebSockets?为什么 LLM 应用几乎都用流式输出？SSE 和 WebSocket 该怎么选？
    Common in ChinaCommon overseasIntermediate#streaming#protocol

    How to reason about it · think before answering

    1. The first half tests latency literacy: separate time-to-first-token from total latency and tie it to sequential generation.
    2. Translate to product terms: feedback within a second versus twenty seconds of blank screen.
    3. For the second half, skip the pros-and-cons table and ask whether the client needs frequent upstream messages.
    4. Server-to-client tokens only means SSE suffices: plain HTTP, proxy-friendly, with built-in reconnection. Voice, collaboration or frequent interrupts justify WebSockets.
    5. State the common shape: plain POST for the request, SSE for the reply, plus a cancel endpoint — which sets up the trap that POST-based SSE cannot use EventSource auto-reconnect.

    分析过程 · 先想清楚再作答

    1. 第一问考的是对延迟指标的敏感度：要能区分「首字延迟」和「全文延迟」，并说出模型逐 token 生成决定了前者远小于后者。
    2. 把它翻译成产品语言：用户 1 秒内看到反馈 vs 对着空白等 20 秒，这是体验的分水岭，不是锦上添花。
    3. 第二问不要背优缺点表，先问自己「客户端需不需要频繁上行」——这一条几乎决定了答案。
    4. 只需要服务器往下推 token，SSE 就够：它跑在普通 HTTP 上，代理和负载均衡友好，还自带重连。需要语音、协同、频繁打断这类双向高频交互，才值得上 WebSocket。
    5. 给出多数产品的真实形态：请求走普通 POST，回复走 SSE，另配一个取消接口——顺势可以引到「POST 的 SSE 用不了 EventSource 的自动重连」这个坑。

    Key points

    • Models emit tokens sequentially; time-to-first-token is far lower than full latency
    • SSE is one-way over HTTP with built-in reconnect and easy proxying, ideal for server→client token streams
    • WebSockets are bidirectional, better when the client sends often (voice, collaboration, interrupts) but harder to load-balance
    • Most chat products: plain POST for the request, SSE for the reply, plus a cancel endpoint

    答题要点

    • 模型逐 token 生成，首字延迟远小于全文延迟；流式让用户 1 秒内看到反馈而不是等 20 秒
    • SSE 是单向、基于 HTTP 的文本协议，自动重连、穿透代理容易，天然适合服务器→客户端的 token 流
    • WebSocket 双向、更适合需要客户端频繁上行（语音、协同编辑、打断）的场景，但代理/负载均衡更麻烦
    • 多数聊天产品：请求用普通 HTTP POST，回复用 SSE；需要打断时再加一个取消接口
  • What fundamentally separates a chatbot from an agent?聊天机器人和 Agent 的本质区别是什么？
    Common in ChinaCommon overseasBasic#agent-basics

    How to reason about it · think before answering

    1. This one invites marketing language; the test is whether your answer names engineering costs.
    2. Give the structure first: a chatbot is one call, an agent loops think → act → observe until the goal is met.
    3. Name the three additions — loop, tools, memory — and stress that tools cause side effects on the world.
    4. Immediately pair each with its cost: permissions and sandboxing, step and budget caps, observability and retries.
    5. Close with a concrete example and the infrastructure it implies: queues, state machines, cost metering.

    分析过程 · 先想清楚再作答

    1. 这题最容易答成营销话术。判断标准很简单：你的回答里有没有出现「工程代价」，没有就是背概念。
    2. 先给结构：聊天是一问一答的单次调用；Agent 是在循环里反复「思考 → 调工具 → 观察」直到目标达成。
    3. 点出三个新增件——循环、工具、记忆——并强调关键差异是「工具能对外部世界产生副作用」，这是可逆与不可逆的分界线。
    4. 紧接着说代价：有副作用就要管权限与沙箱，有循环就要管步数与成本预算，有多步就要可观测性和失败重试。这一段才是面试官想听的。
    5. 用一个具体例子收尾（能查库、发消息、定时提醒的助手），并点出它背后需要队列、状态机、成本计量。

    Key points

    • A chatbot answers once; an agent loops think → act (tool call) → observe until the goal is met
    • Three additions: a loop (multi-step), tools (side effects on the world), memory (across turns/sessions)
    • They bring engineering concerns: tool permissions and sandboxing, retries, step/cost budgets, observability
    • Example: an assistant that queries a DB, sends messages and schedules reminders needs queues, state machines and cost tracking

    答题要点

    • 聊天机器人是一问一答；Agent 是模型在一个循环里反复思考、调用工具、观察结果直到完成目标
    • 三个新增件：循环（多步）、工具（能对外界产生副作用）、记忆（跨轮次/跨会话）
    • 随之而来的工程问题：工具权限与沙箱、失败重试、成本与步数预算、可观测性
    • 举例：一个能查库、发消息、定时提醒的助手，背后要有消息队列、状态机和成本计量
  • What do temperature and top_p control, and when would you use 0 versus 0.7?temperature 和 top_p 分别控制什么？什么场景用 0，什么场景用 0.7？
    Common overseasBasic#llm-basics#sampling

    How to reason about it · think before answering

    1. Establish that both act on the same next-token distribution but in different ways — that is the discriminator.
    2. temperature rescales the whole distribution; top_p truncates it to the smallest set reaching cumulative probability p.
    3. Hence the practical rule: tune one, not both, or you cannot attribute a regression.
    4. Choose by reproducibility, not by vibes: tool arguments, classification and structured output must be reproducible, so use 0.
    5. Add the agent angle: planning and tool-calling steps stay cold; only the final user-facing prose warrants higher values.

    分析过程 · 先想清楚再作答

    1. 先说清两者作用在同一个地方——模型算出的下一个 token 概率分布——但作用方式不同，这是区分度所在。
    2. temperature 是缩放整个分布：越低越尖锐、越确定；top_p 是截断——只保留累计概率达到 p 的那一小圈候选再采样。
    3. 由此推出实践建议：一般只调其中一个，两个同时调会互相干扰，出了问题分不清是谁造成的。
    4. 选值不按「创意程度」凭感觉，按「这一步的输出要不要可复现」来定：工具参数、分类判断、结构化输出必须可复现，用 0。
    5. 补一句 Agent 视角：Agent 的规划与工具调用环节几乎都用低温，只有最终面向用户的自然语言回复才考虑调高。

    Key points

    • temperature rescales the next-token distribution: lower is more deterministic, higher more random
    • top_p samples only from the smallest set whose cumulative probability reaches p; tune one, not both
    • Use ~0 for structured output, tool arguments and classification to keep results reproducible
    • Use 0.7–1.0 for creative writing; planning steps in production agents usually stay low

    答题要点

    • temperature 缩放下一个 token 的概率分布：越低越确定，越高越随机
    • top_p 只从累计概率达到 p 的候选里采样，是另一种截断随机性的方式；一般只调其中一个
    • 结构化输出、工具参数、分类判断用 0 或接近 0，保证可复现
    • 创意写作、头脑风暴用 0.7–1.0；生产 Agent 的规划步骤通常也偏低温
  • A streaming reply is cut off mid-way. What do the client and server each do, and can EventSource auto-reconnect help?流式回复到一半网络断了，前端和后端各要做什么？EventSource 的自动重连能用上吗？
    Common in ChinaCommon overseasIntermediate#streaming#reliability#sse

    How to reason about it · think before answering

    1. The trap is the second half: people who memorized 'SSE reconnects automatically' answer yes, which is wrong.
    2. Native EventSource does auto-reconnect per spec, sending Last-Event-ID, with the server marking events via id: and setting the interval via retry: — but it only issues GET and requires Content-Type text/event-stream.
    3. LLM chat APIs require POST because messages go in the body, so real clients use fetch plus hand-written SSE parsing, where none of that machinery applies.
    4. So the client owns detection, retry and buffering of what arrived; the server's job is making retries safe — resumable output and idempotent side effects.
    5. Give the continuation strategy and its limits: feed the received prefix back as context, but tool-use and thinking blocks cannot be partially recovered — resume from the last complete text block.
    6. Follow-up to expect: does a non-200 reconnect? Per spec no — a non-200 status or wrong Content-Type fails the connection, and a 204 tells the browser to stop reconnecting.

    分析过程 · 先想清楚再作答

    1. 这题的陷阱在后半句。很多人背过「SSE 自带重连」，就直接答自动重连能救——那是错的，必须先分清两种 SSE 用法。
    2. 浏览器原生 EventSource 确实按规范自动重连：重连时带 Last-Event-ID 请求头，服务器用 id: 打点、用 retry: 设间隔；但它只能发 GET，且要求响应 Content-Type 是 text/event-stream。
    3. 而 LLM chat API 必须 POST（messages 要放在请求体里），所以实际用的是 fetch 加手写 SSE 解析——EventSource 那套自动重连一行都用不上。
    4. 于是前端职责变成：自己判定断流、自己重试、自己保存已收到的部分。后端职责是让重试是安全的——响应可续、副作用幂等。
    5. 给出续写策略并说清边界：把已收到的内容作为上下文构造续写请求；但工具调用块和思考块无法部分恢复，只能从最近的完整文本块续。
    6. 可预期追问：非 200 响应会重连吗？按规范不会——状态码不是 200 或 Content-Type 不对，连接直接判定失败；服务器还可以用 204 主动叫停重连。

    Key points

    • Separate the two SSE modes: native EventSource auto-reconnects with Last-Event-ID but is GET-only; LLM APIs use POST and cannot rely on it
    • The client must therefore detect the break, retry itself, and keep whatever text already arrived
    • Continuation: send the received prefix as context so the model resumes rather than restarting the turn
    • Limits: tool_use and thinking blocks cannot be partially recovered; resume from the last complete text block
    • The server must make retries safe: resumable responses, idempotent tool side effects, correct billing for tokens already produced

    答题要点

    • 先区分两种 SSE：浏览器原生 EventSource 自动重连并带 Last-Event-ID，但只能 GET；LLM API 走 POST，用不上这套
    • 所以前端要自己检测断流、自己重试，并保留已收到的部分内容
    • 续写策略：把已收到的内容作为上下文发起新请求，让模型接着写，而不是整轮重来
    • 边界：tool_use 和 thinking 块无法部分恢复，只能从最近的完整文本块续
    • 后端要保证重试安全：响应可续、工具副作用幂等，并对已产生的用量正确计费
  • The user backgrounds the app or closes the tab. How do you restore a reply that was still being generated?用户切到后台或者直接关掉网页，回来后怎么恢复那条还在生成的回复？
    Common in ChinaCommon overseasDeep dive#streaming#reliability#architecture

    How to reason about it · think before answering

    1. First separate this from a dropped connection: the client is gone, so no client-side retry will ever run.
    2. That leaves one option — the generation must outlive the client, which means persisting the stream server-side.
    3. Concretely: assign a stream id per generation; the server pushes tokens to the live connection while also writing them to storage such as Redis, and the chat record stores that activeStreamId.
    4. Recovery is a separate GET endpoint: the client asks with the chat id, the server locates the stream by activeStreamId and resumes; with no active stream it returns 204.
    5. Name the costs, not just the design: extra storage, expiry/cleanup, and concurrency when several connections consume the same stream.
    6. Extension: this differs from ordinary message persistence because the reply is still being produced — you need a resumable stream, not a static row.

    分析过程 · 先想清楚再作答

    1. 先识别这题和「网络断了」不是同一个问题：客户端已经不存在了，任何写在前端的重试逻辑都不会执行。
    2. 由此推出唯一出路：生成过程必须能脱离这个客户端独立存活，也就是把流本身放到服务端持久化。
    3. 落到具体架构：发起请求时给这轮生成分配一个流 id，服务端一边把 token 推给当前连接，一边把同样的内容写进 Redis 之类的存储；会话记录里保存这个 activeStreamId。
    4. 恢复路径是另开一个 GET 端点：客户端带着会话 id 请求，服务端按 activeStreamId 找到那条流并接着推；找不到活跃流就返回 204，让前端知道没有需要恢复的东西。
    5. 说清代价，别只说方案：多了一份存储、一套过期清理、以及「同一条流可能被多个连接消费」的并发问题。
    6. 延伸：这套结构和普通聊天产品的「消息已持久化，重进会话直接读库」不同——区别在于回复还在生成中，需要的是可续的流而不是一条静态记录。

    Key points

    • The client is gone, so recovery must live server-side: the generation has to outlive the connection
    • Assign a stream id at start; the server writes tokens to Redis while streaming, and the chat stores activeStreamId
    • Resume through a dedicated GET endpoint that replays the active stream, returning 204 when there is none
    • Costs: extra storage, expiry and cleanup, and concurrent consumers of one stream
    • It differs from plain message persistence because the reply is still in flight, so you need a resumable stream

    答题要点

    • 客户端已经不在了，前端重试无从谈起，必须让生成过程在服务端独立存活
    • 发起生成时分配流 id，服务端边推送边把内容写进 Redis，会话里记录 activeStreamId
    • 恢复走单独的 GET 端点：按会话 id 找到活跃流接着推，没有活跃流就返回 204
    • 代价：额外存储、过期清理，以及同一条流被多个连接消费的并发处理
    • 与「消息持久化后重新读库」的区别在于回复仍在生成中，需要的是可续的流
  • After a retry, how do you avoid double billing and re-executing tool calls that already ran?断线重试之后，怎么保证不重复计费、也不重复执行已经做过的工具调用？
    Common in ChinaCommon overseasDeep dive#reliability#tools#idempotency

    How to reason about it · think before answering

    1. Split it in two: billing is a bookkeeping problem, tool side effects are an execution problem, and they have different fixes.
    2. Billing: meter server-side by tokens actually produced, not by request count. Tokens produced before the break are real cost; so are retry tokens. The point is not to count the same batch twice.
    3. That needs a stable identifier: give each generation a run id and dedupe usage records by run id plus sequence.
    4. Tools: the danger is side-effecting tools — transfers, messages, orders. The fix is an idempotency key derived from the call arguments, checked before execution.
    5. Add the state-machine view: record each call as pending / running / done and replay only what is unfinished.
    6. Follow-up: who generates the idempotency key? The caller must, and pass it along — a server-generated key cannot stay stable across retries.

    分析过程 · 先想清楚再作答

    1. 先把问题拆成两半：计费是「记录问题」，工具副作用是「执行问题」，两者的解法不同，混在一起答会含糊。
    2. 计费侧：用量应该在服务端按实际收到的 token 记账，而不是按「请求次数」。断在中途已经产生的 token 是真实成本，要照记；重试产生的是新成本，也要照记——关键是别把同一批 token 记两遍。
    3. 为此需要一个稳定的标识：给每轮生成一个 run id，用量记录以 run id + 序号去重，重放同一段不会重复入账。
    4. 工具侧：真正危险的是有副作用的工具（转账、发消息、下单）。解法是幂等键——由调用参数派生一个稳定的 key，执行前先查这个 key 是否已有结果，有就直接返回旧结果。
    5. 补一层状态机视角：把每次工具调用记为「待执行 / 执行中 / 已完成」，重试时只重放未完成的部分，已完成的直接取结果，这也是恢复中断任务的通用做法。
    6. 常见追问：幂等键该谁生成？应由客户端或调度侧生成并随请求传递，服务端自己生成就没法跨重试保持一致。

    Key points

    • Separate billing (bookkeeping) from tool side effects (execution); they need different mechanisms
    • Meter by tokens actually produced, deduped by run id plus sequence so one batch is never counted twice
    • Guard side-effecting tools with an idempotency key derived from the call arguments
    • Model each tool call as pending / running / done and replay only unfinished work
    • The caller must generate and pass the idempotency key so it stays stable across retries

    答题要点

    • 拆成两个问题：计费是记账问题，工具副作用是执行问题，解法不同
    • 计费按服务端实际产生的 token 记，用 run id 加序号去重，避免同一批 token 重复入账
    • 有副作用的工具用幂等键：由调用参数派生稳定 key，执行前先查是否已有结果
    • 把每次工具调用记成待执行/执行中/已完成的状态机，重试只重放未完成的部分
    • 幂等键要由调用方生成并随请求传递，服务端自行生成无法跨重试保持一致
  • On mobile, connectivity is flaky. How would you design the reconnection strategy for a chat feature?移动端 App 里的对话，网络频繁抖动，你会怎么设计重连策略？
    Common in ChinaCommon overseasIntermediate#reliability#mobile#streaming

    How to reason about it · think before answering

    1. Start with what makes mobile different: network switches between WiFi and cellular, the OS suspends apps, background time is limited.
    2. Use exponential backoff with jitter; jitter is the commonly missed part that prevents a thundering herd when a wide outage clears.
    3. Set ceilings: max attempts and max interval, then surface an explicit reload action instead of retrying silently forever.
    4. Distinguish a brief blip from being genuinely offline: subscribe to OS connectivity events, stop retrying when offline, and reconnect on the restore event — far cheaper on battery than blind timers.
    5. Combine with server-side persistence: after the OS kills the app, resume by chat id rather than reconstructing from local cache.
    6. Finally the send path: queue outgoing messages while offline and replay them in order, each with an idempotency key.

    分析过程 · 先想清楚再作答

    1. 先说明移动端和浏览器的差别：网络在 WiFi 与蜂窝之间切换、App 会被系统挂起、后台执行时间受限，所以不能照搬网页那套。
    2. 重试节奏用指数退避加随机抖动。抖动这一条常被忽略，但它是防止大面积断网恢复后所有客户端同时涌上来把服务打垮的关键。
    3. 要设上限：最大重试次数与最大退避间隔，超过就转成显式的「重新加载」按钮交给用户，而不是无限静默重试。
    4. 区分「短暂抖动」和「真的没网」：监听系统的网络状态变化，没网时直接停止重试并进入离线态，等网络恢复事件再立刻重连，比盲目定时重试省电得多。
    5. 结合上一题的服务端持久化：App 被系统杀掉后重进，靠会话 id 请求恢复端点，而不是指望本地缓存拼出完整回复。
    6. 最后补发送侧：用户在离线时发出的消息进本地队列，恢复后按序重发，且每条带幂等键，避免重复发送。

    Key points

    • Mobile differs: network handoffs, OS suspension, limited background time — do not copy the web strategy
    • Exponential backoff with jitter, where jitter prevents a reconnect storm when an outage clears
    • Cap attempts and interval, then hand the user an explicit reload instead of retrying forever
    • Listen to OS connectivity events: stop while offline, reconnect on restore, which saves battery over polling
    • Resume replies via server-side persistence by chat id; queue outgoing messages with idempotency keys

    答题要点

    • 移动端特殊性：WiFi 与蜂窝切换、App 被挂起、后台执行时间受限，不能照搬网页策略
    • 指数退避加随机抖动，抖动用于避免大面积恢复时的重连风暴
    • 设最大重试次数与最大间隔，超过后转为显式的重新加载入口，不做无限静默重试
    • 监听系统网络状态：离线直接停重试进入离线态，收到恢复事件再重连，比定时轮询省电
    • 回复恢复依赖服务端持久化，靠会话 id 请求恢复端点；发送侧用本地队列加幂等键按序重发
  • A user pressing stop and a dropped connection both look like a closed connection server-side. How do you tell them apart?用户主动点「停止生成」和网络意外断开，在服务端看起来都是连接没了，怎么区分处理？
    Common in ChinaCommon overseasDeep dive#streaming#reliability#ux

    How to reason about it · think before answering

    1. Say why it matters: stop means the user no longer wants the output, so free compute and end the run; a drop means they still want it, so preserve the result for resumption.
    2. Connection state alone cannot distinguish them — it looks identical — so you need an explicit signal.
    3. Give stop its own endpoint: the client calls it with the run id before closing, and the server marks the run as user-cancelled and aborts the upstream call.
    4. Treat a bare connection close as an unexpected drop: keep persisting output and hold the stream for resumption.
    5. Add the real-world caveat: the stop request itself may fail to send when the network is down, so the server needs a fallback — end a stream with no consumer after a timeout.
    6. Extend to billing: both cases still owe for tokens already produced, since the upstream provider has charged; they differ only in whether output is retained.

    分析过程 · 先想清楚再作答

    1. 先点破为什么要区分：主动停止是「用户不想要了」，应当立即释放算力并结束这轮；意外断开是「用户还想要」，理应保留结果供恢复。处理反了，用户要么白花钱，要么回来发现内容没了。
    2. 所以不能只靠 TCP 连接状态判断——它对两种情况的表现是一样的。必须有一个显式信号。
    3. 做法是给「停止」单独一个接口：前端点停止时先调这个接口，带上 run id，服务端据此把该轮标记为「用户取消」，再中止上游模型调用。
    4. 而单纯的连接关闭一律按「意外断开」处理：继续把已生成内容落盘、保留可恢复的流，等客户端回来续。
    5. 补一个现实约束：停止请求本身也可能因为断网而发不出去。所以服务端还需要兜底——比如流没有任何消费者超过一定时间就自行结束，避免算力空转。
    6. 延伸到计费：两种情况都要为已经产生的 token 计费，因为上游厂商已经收了钱；区别只在于要不要保留结果和是否继续生成。

    Key points

    • The semantics are opposite: stop frees compute immediately, a drop preserves output for resumption
    • Connection state cannot distinguish them, so add an explicit stop endpoint carrying the run id
    • Treat a bare close as an unexpected drop: keep persisting and hold the stream for resume
    • Fallback: the stop call may itself fail to send, so end streams with no consumer after a timeout
    • Both still bill for tokens already produced; they differ only in retention and whether generation continues

    答题要点

    • 两者语义相反：主动停止要立即释放算力并结束，意外断开要保留结果等待恢复
    • TCP 连接状态无法区分，必须有显式信号：给停止单独一个接口，带 run id 标记为用户取消
    • 只收到连接关闭一律按意外断开处理，继续落盘并保留可恢复的流
    • 兜底：停止请求本身也可能发不出去，服务端需对长时间无消费者的流自行结束
    • 计费上两者都要为已产生的 token 记账，区别只在于是否保留结果、是否继续生成

Comments