Dayward AI
中文
Week 2 · D14About 6 hours

Deployment and Operations: Multi-Worker Compose, Heartbeats, Health Checks, Graceful Shutdown, Dev/Prod Isolation; Week Two Retrospective

Run mini-koda as a multi-worker deployment with docker compose, add heartbeats, health checks, and graceful shutdown, sort out how dev and prod stay isolated, and wrap up week two.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Scale the worker to multiple instances with docker compose and observe how traffic is split
  2. Implement worker heartbeat reporting and a gateway health-check endpoint
  3. Implement graceful shutdown: finish the task in hand before exiting once a stop signal arrives

Yesterday closed on this: the scheduler, the bus, the worker, the database, and the ledger all run in one terminal on your machine, and you find problems by reading logs. Today it becomes a deployment somebody else could take over, and then we string week two into one line.

Plain-Language Walkthrough

One image, three shop assistants: multi-replica workers in compose

The convenience store downstairs is open around the clock, and not because of any one assistant — one set of procedures, three shifts rotating, and whoever is on can take payments, restock, and count inventory. A shop is a role plus a rota, not a person. Define the role clearly and people become replaceable.

D8 through D13 already defined the roles: the gateway is stateless, the worker takes work off the bus, and all state lives in Postgres and Redis. Scaling replicas was architecturally finished long ago; today only declares it.

The declaration lives in a compose file. Its division of labor differs from D7's Dockerfile: a Dockerfile answers how this one process starts, and compose answers how these processes assemble into a system. D7's image does not change one line today.

YAMLYAML
services:
  gateway:
    build: .
    environment:
      ROLE: gateway            # one image, two roles, branching on an environment variable
      APP_ENV: dev
      REDIS_URL: redis://redis:6379
    ports: ['3014:3014']       # only this one needs a port exposed
    depends_on:
      postgres: { condition: service_healthy }
      redis: { condition: service_healthy }
    stop_grace_period: 25s
 
  worker:
    build: .                   # the same image
    environment:
      ROLE: worker
      APP_ENV: dev
      REDIS_URL: redis://redis:6379
    deploy:
      replicas: 3              # three replicas, each its own container and hostname
    stop_grace_period: 25s     # must exceed the 20-second grace in the code, see section four

Four decisions deserve explanation.

One image, not two. The gateway and the worker already share a repository, a schema, and configuration. Two images means two pipelines and two version numbers, and produces the hardest failure to diagnose: the gateway on the new version with the worker still on the old. Branch on one environment variable and versions align by construction.

Worker replicas need no port mapping. They accept no inbound connections and go to the bus for work. So adding a replica touches no load balancer, no service registry, and no routing — a new container comes up, issues an XREADGROUP, and has work; whereas every extra gateway must be attached behind a load balancer. Pull scales more easily than push, a downstream benefit of D9's choice of a message bus.

Replica names need thought. Three replicas share one set of environment variables, so the consumer name cannot live there. This course uses the container hostname, which is naturally one per container. The cost is the trap D9 named: rebuilding a container changes the name, unacknowledged messages under the old one become orphans, and only XAUTOCLAIM recovers them.

The replica count has a ceiling, and it is not CPU. Connections equal replicas times pool size, and Postgres's default max_connections is only 100 — 10 replicas with 10 connections each saturate it, and the eleventh gets "too many clients." Count connections before scaling; it is the first wall you hit going from three replicas to thirty, and the error has nothing to do with your business code.

At this point docker compose up -d brings up a multi-replica deployment and docker compose ps faithfully shows three Up. But Up says only that the process exists — it may be stuck in a garbage collection doing nothing. And the next release replaces all three containers, so how do you guarantee no in-flight conversation is cut off?

A clock-in record beats watching: heartbeats

A store manager does not watch each assistant all day, they check the clock-in record: only somebody who has missed three in a row gets a visit. Not constant watching, but an agreed rhythm, with an alert when the rhythm breaks.

That rhythm is the heartbeat, and this course's convention is reporting every 5 seconds, with three missed cycles (15 seconds) counting as lost.

Why is it needed? Because "the process exists" and "it is still working" differ. An orchestrator sees the former: a process that has not exited is Up. But a worker can be alive with all work stopped — an infinite loop occupying the event loop, an exhausted connection pool timing out every fetch, a noisy neighbor saturating the CPU. That kind of zombie is the most common failure shape in production, and precisely the one an orchestrator cannot see.

Get the direction right: a heartbeat is pushed by the replica, not polled by the gateway. Containers change IP and hostname at will, so a poller needs a forever-changing list — and maintaining that list itself requires heartbeats, which is circular. Pushing needs one place both sides know, and this course uses a Redis hash keyed by replica name.

Do not report only "I am alive." Three items each earn their place: a timestamp for liveness, how many tasks are in flight to distinguish idle from overloaded, and a version number to tell you during a rolling release how many replicas of each generation remain.

heartbeat.js
const INTERVAL_MS = 5_000
const STALE_MS = 15_000 // 3 reporting cycles
 
export function startHeartbeat(redis, key, workerId, inFlight) {
  const beat = () =>
    redis.hset(
      key,
      workerId,
      JSON.stringify({ beatAt: Date.now(), inFlight: inFlight(), version: VERSION })
    )
  void beat() // beat immediately so the dashboard does not miss a cycle showing a new replica
  const timer = setInterval(beat, INTERVAL_MS)
  timer.unref() // the heartbeat must never be why the process will not exit
  return () => clearInterval(timer)
}
 
// Liveness is one subtraction. The threshold is 3 cycles rather than 1: one GC pause can
// delay a report, and too tight a threshold makes the dashboard flicker red and green
// until the on-call engineer stops looking at it two days later.
export const isAlive = (row, now) => now - row.beatAt <= STALE_MS

Two engineering costs. Write volume: 3 replicas every 5 seconds is about 52,000 writes a day, negligible; at 1 second and 100 replicas it is 8.64 million a day, which needs pricing. And the shape of the storage: separate keys with expiries are simpler, since a missing key means lost — but the last-heartbeat time goes with the key, and the dashboard can then only show "absent" rather than "lost 23 seconds ago," which is what you want while investigating. This course chooses a hash plus a timestamp, at the cost of dead rows to clean up.

Head office's open sign: what a health check should answer

A convenience-store chain's head office has an open-status board. Note that one branch closing does not close the others. The board is for head office, not a precondition for each shop's door.

That is where health checks most often go wrong — one word answers three different questions, and merging them into one endpoint causes incidents:

  • Liveness: does this process need restarting?
  • Readiness: may traffic be sent right now?
  • A dependency dashboard: what is the cluster's state?

D8 covered the difference between the first two and left a warning: do not check every downstream dependency in a health check. Today gives it a concrete shape — the readiness probe checks only what the gateway itself requires (a select 1), and the heartbeat dashboard is a separate endpoint.

JSONJSON
// GET /admin/workers - the dashboard, for on-call engineers and alert rules
{
  "status": "degraded",
  "total": 3,
  "alive": 2,
  "stale": 1,
  "workers": [
    { "workerId": "worker-a", "alive": true, "lastBeatMs": 1200, "inFlight": 2, "version": "v2" },
    { "workerId": "worker-b", "alive": true, "lastBeatMs": 3400, "inFlight": 0, "version": "v2" },
    { "workerId": "worker-c", "alive": false, "lastBeatMs": 41000, "inFlight": 1, "version": "v1" }
  ]
}

What happens if you merge the two? One worker goes missing, every gateway's readiness probe goes red at once, and the orchestrator pulls the whole intake layer out — a non-critical failure escalated into total unavailability. And that missing worker does not affect intake at all: messages are still in the stream, unacknowledged ones get claimed (D9), and its lease changes hands on TTL expiry (D10). Intake should keep taking work.

So the answer to "how does the gateway decide whether a worker is usable" is slightly counterintuitive: it does not, and it does not need to. The gateway never assigns work to a specific worker; assignment is decided by the consumer group and leases (D9, D10). Heartbeats are for observation and alerting, not routing. Grasp that and you will not write "deliver to the least busy worker" — which would move dispatch responsibility back into the gateway and glue D8's two layers together again.

How does the dashboard drive alerts? Two rules. Alert when the live replica count is below expectation — declaring 3 and seeing only 2 clocked in means one has neither exited nor is working, exactly the failure only heartbeats can see. A red dashboard does not automatically pull traffic, it summons a person: "lost" is sometimes just Redis wobbling, and automatic action turns a self-healing wobble into a real outage.

You cannot abandon a customer mid-transaction at shift change: graceful shutdown

Today's most important section. The convenience store's handover rule is plain: the transaction in progress must be completed. The new assistant may start serving new customers immediately, and the one already begun cannot be abandoned.

Processes are the same. Releases, scale-downs, maintenance, and preemptible reclamation all send SIGTERM, wait a grace period, and SIGKILL on timeout. SIGKILL cannot be blocked, and landing it on a worker mid-agent-loop has concrete consequences: that run stays at running forever with the user spinning; the model was paid for and the reply never persisted; and the unacknowledged message waits out the idle threshold (D9). One release cutting off dozens of conversations is the everyday cost of no graceful shutdown.

After SIGTERM there are three steps, and the order is fixed:

TextText
t=0.0s   SIGTERM received
         Step one: refuse new work - flip the switch so the consume loop stops reading
         from the stream on its next round. Messages already read but not started stay in
         pending for somebody else to claim, which is faster than forcing a whole batch through
t=0.0s   Step two: wait for the execution in hand, but at most 20 seconds
t=3.7s   the run in hand wraps up as done and its message is acknowledged
         Step three: actively return the lease and deregister from the heartbeat dashboard
t=3.8s   the process exits (16 seconds of grace period unused)

Those 20 seconds in step two are this course's fixed convention, derived from the upper bound of one normal execution plus headroom. The wait must have a ceiling: one hung model call means waiting forever, and the grace period ends in SIGKILL regardless — better to concede and exit than to be cut down, since the unacknowledged message is still in pending for somebody to redo.

Step three is the hook D10 deliberately left for today. D10's leases change hands naturally on TTL expiry, and a kill -9 leaves no chance to return one, so the successor waits up to a full TTL (30 seconds). But a planned shutdown should not take that road — you know you are leaving, so why make the next shift wait? Return it actively and the successor starts on its next sweep; likewise deregister actively so that a planned shutdown and a dead process are two distinct dashboard phenomena rather than both showing as lost for 15 seconds.

Returning must carry a condition: delete only the badge that still bears your own name. If the lease has expired and somebody just claimed it, an unconditional delete tears up their badge — the criterion is D10's renewal rule, with comparison and modification in one step.

shutdown.js
async function shutdown() {
  draining = true // 1. refuse new: the consume loop stops reading on its next round
  stopHeartbeat()
 
  // 2. Wait for what is in hand, at most 20 seconds. The await chain inside the consume
  // loop runs all the way into the execution function, so waiting for the loop to end
  // is waiting for the work in hand.
  const timeout = new Promise((r) => setTimeout(() => r('timeout'), GRACE_MS).unref())
  if ((await Promise.race([loop, timeout])) === 'timeout') {
    log.warn({ inFlight: inFlight.size }, 'still unfinished after the grace period, giving up')
  }
 
  // 3. Return the lease actively: conditional release, deleting only the badge with my name
  for (const shard of held) await lease.releaseIfOwner(shard, workerId)
  await presence.forget(workerId)
}
 
// Two SIGTERMs should not start two drains, so remember this shutdown's promise
let stopping
process.on('SIGTERM', () => {
  stopping ??= shutdown().then(() => process.exit(0))
})

The differences between the four say a lot about each language's character: Java's ExecutorService is already a refuse-new, await, force-interrupt triple; Python's asyncio.wait_for packages "wait with a ceiling" into one function; Swift relies on structured concurrency's cancellation propagation; and only Node assembles a timeout race by hand. The semantics of the three steps are identical, and the language only decides how many lines you write.

Two accompanying items, and missing either voids everything above.

One, the configured grace period must exceed the wait ceiling in the code. Code waiting 20 seconds against compose's default 10 means SIGKILL at second 10 and the three steps never getting past halfway. It is stop_grace_period in compose and terminationGracePeriodSeconds in Kubernetes, and both need headroom (25 seconds here). After writing shutdown logic, the first thing to do is check that setting.

Two, the signal has to actually reach your process. D7 covered it: make the start command a package manager and PID 1 is the package manager rather than your process, SIGTERM may never arrive, and the shutdown code never runs once. Start the business process directly in exec form.

Put those two together with the three steps and you have the complete answer to rolling releases that do not cut off work: the orchestrator removes traffic, sends SIGTERM, and waits out the grace period; the process finishes the work in hand, returns ownership, and exits cleanly. Asked in an interview how a rolling release avoids interrupting in-flight tasks, that sentence is the skeleton of the answer.

One codebase, two worlds: isolating dev from prod

The convenience store's staff training uses a separate till running the same software, and it must never connect to real inventory and real accounts — no training keystroke may become a real transaction.

Local development is that training till. Same code, often the same Redis, and the easiest incident is this: you start a worker locally to debug, it connects to the production stream, and it takes and executes real users' messages. That kind of incident raises no error, and both sides' logs read "all normal" — from the code's point of view it did dutifully process a message.

Isolate in layers, cheapest first: namespacing (key prefixes), separate instances (their own Redis and database), separate environments (network, credentials, accounts). Production eventually needs the third; this course focuses on the first, the cheapest and the easiest to leave incomplete.

This course's convention: every key name carries a dev: or prod: prefix taken from APP_ENV. D9's input stream therefore becomes dev:koda:runs or prod:koda:runs, and the same goes for lease keys and the heartbeat hash. The crucial part is that the prefix is assembled in exactly one function — scattered around, missing one of twenty key names is the same as no isolation, and the one you miss is usually the newest and least tested feature.

env.js
export function appEnv() {
  const raw = process.env.APP_ENV ?? 'dev'
  // Only two values are accepted: typing development errors immediately rather than
  // silently becoming a third environment
  if (raw !== 'dev' && raw !== 'prod') throw new Error(`APP_ENV must be dev or prod, got ${raw}`)
  return raw
}
 
// Every key name goes through this layer, and the prefix is assembled here only
export const namespaced = (name) => `${appEnv()}:${name}`
export const runsStream = () => namespaced('koda:runs')
 
// Forbid the in-memory implementation in production: without this line, one configuration
// slip (forgetting to inject REDIS_URL) lets production processes come up quietly, each
// working in its own memory, with every health check green
export function assertProdSafety(mode) {
  if (appEnv() === 'prod' && mode !== 'real') throw new Error('the in-memory implementation is not allowed in production')
}

Beyond prefixes, three things must accompany it. Split credentials: the local key reaches only the development database. Destructive operations must check the environment: scripts that truncate a database, replay dead letters, or reindex read APP_ENV first and require explicit confirmation in production. Forbid degraded implementations in production: if a configuration slip lets production reach the in-memory one, processes come up quietly, each working in its own memory, with health checks all green — a failure that hides for hours. Erroring out at startup is far cheaper.

W2 retrospective: a process taken apart, grown into a system

Looking back over seven days, this was not seven middleware tutorials but one single-process service taken apart and regrown as a distributed system, each day forced by the one before:

  • D8 replaced the session Map with three tables and split the service into intake and execution. The idempotency_key unique constraint is the final arbiter of idempotency — check-then-insert is not idempotency.
  • D9 filled that empty middle box with Redis Streams. At-least-once is the default semantics and exactly-once is an effect the consumer's idempotency produces; the source of truth is always the runs table and the stream is only a trigger.
  • D10 noted that a consumer group's unit of assignment is one message while the business's serial unit is one user, producing sharding plus leases. Renewal must be atomic, which is the entire reason for Lua.
  • D11 connected the output back to the user: the state machine blocks illegal transitions, fragments return numbered, and reconnection rests on the last received number plus one. The output stream uses a broadcast read and the input stream a consumer group.
  • D12 fitted long-term memory and drew the line against D6 in its first paragraph: one solves "this round will not fit," the other "cannot recall last month." What costs money is not the embedding but the context the results occupy.
  • D13 went from passive to active: a scheduled task only changes who presses the button from the user to a clock, with the idempotency key anchored to the scheduled minute.
  • D14, today: multiple replicas, heartbeats making zombies visible, probes separating "may I take traffic" from "what is the cluster's state," graceful shutdown so releases do not cut off work, and environment prefixes so local work cannot touch production.

If you take away only three sentences, I suggest these.

One, every step of distribution first removes a guarantee that was free in one process, and then buys it back explicitly. Order, state, identity, exactly-once, even "is this process alive" are all free within one process; across processes each costs code, and each has a clearly priced cost.

Two, every new mechanism needs an answer to "what happens when it fails." Leases split-brain, heartbeats misjudge, shutdowns time out, idempotency keys get the wrong anchor. Unable to name the failure mode means you have not verified it — and in interviews that is the fastest cut between having read about something and having run it in production.

Three, every operational question reduces to one: who can be replaced, and when. The stateless can be killed at any time (D8), the stateful need a handover protocol (today's graceful shutdown), and judging whether to replace something needs an observation surface independent of the process itself (heartbeats). A production-grade IM Agent platform made exactly these layers solid before features had anywhere to grow.

As of today, one agent's production form is complete. But real requirements do not let one agent do everything: support has to triage first, then check stock, then draft a refund proposal, and the proposal needs review. Stuff all those responsibilities into one prompt and it does none of them well. Week three, starting tomorrow, is about making several agents divide the work.

Source Reading

Hands-On Lab

🧪 D14 lab: the full chain at --scale worker=3 plus a heartbeat dashboard

Code location: labs/agent-30days/day-14-compose-multi-worker

Acceptance criteria:

  1. All five self-checks under MOCK=1 SELFTEST=1 pnpm start pass with exit code 0 (starter/ as-is passes only check 1, with the four exercise points mapping to the four failures).
  2. Check 1: 3 replicas share 9 messages from one stream, every replica processes some, and the total is 9 with 9 remaining after deduplication.
  3. Checks 2 and 3: make one replica stop clocking in (simulating a zombie), and after more than 15 seconds (3 reporting cycles) the dashboard shows it lost and the status degraded, while the readiness probe still returns 200.
  4. Check 4: send one replica SIGTERM and the execution in hand still reaches done (not stuck at running), with a wait under the 20-second grace; after shutdown it touches no newly arriving task; and the lease is returned actively so the successor starts in far less than one TTL.
  5. Check 5: dev and prod compute different stream names, and publishing to the prod stream leaves the dev-side probe reading 0.

Under MOCK=1 there are zero external services: the in-memory implementation in src/infra/ is not a stub, with both the lease's TTL-expiry semantics and the consumer group's pending semantics genuinely written out, so lost-replica detection and lease handover are visible offline. The self-check compresses the clock twentyfold (heartbeat 250 ms, lost 750 ms, grace 1 second) in the same proportions as the production values — otherwise you would wait over a minute to see one loss. With Docker, docker compose up -d --build in the lab root. If you get stuck, read the README's common-traps section first.

  1. Run MOCK=1 SELFTEST=1 pnpm start as-is first; the wording of those four failures is your to-do list.
  2. Exercise 1, liveness: make the replica that stopped clocking in count as lost after 3 cycles, taking check 2 from 0 lost to 1.
  3. Exercise 2, dashboard aggregation: compute live and lost counts, degrade the status when one is lost, and confirm the readiness probe still returns 200 — that is a criterion, not an afterthought.
  4. Exercise 3, the three steps of graceful shutdown: refuse new, wait for what is in hand (20-second ceiling), conditionally return the lease and deregister the heartbeat, and check 4's run finishes as done rather than running.
  5. Exercise 4, environment prefixes. Then submit a 6-second slow task and immediately docker compose stop worker, and watch it wait for the task before exiting.

Interview Questions

Today's five questions are in the bank below; the first four cover heartbeats and health checks, graceful shutdown, rolling releases, and environment isolation, and the last is a full system-design question: design an IM Agent platform. That question's analysis gives a complete answer framework, and walking through this week against it is worth more than ten smaller questions. Expand a question and read the analysis before the key points.

Checklist and Tomorrow

  • Scale the worker to multiple instances with docker compose and observe how traffic is split
  • Implement worker heartbeat reporting and a gateway health-check endpoint
  • Implement graceful shutdown: finish the task in hand before exiting once a stop signal arrives
  • Say what the readiness probe, the liveness probe, and the heartbeat dashboard each answer, and what incident merging them causes
  • Name the three steps of graceful shutdown, and explain why the grace period must exceed the wait ceiling in the code
  • Without notes, give one sentence per day from D8 to D14, saying which problem from the previous day forced it
  • All 5 acceptance criteria of the lab pass (all five self-checks green)
  • Answer at least 4 of the 5 interview questions, and talk for a full 20 minutes on the system-design one

Tomorrow (D15) begins week three, turning one agent into a group of them: what Router and Supervisor, Planner-Executor, Critic, Swarm, and Blackboard each look like, when multiple agents solve a problem and when they create one, and then a first three-node graph with LangGraph.js. Why after deployment? Because multiple agents multiply everything from today by a factor — more steps, more state, more money. Get one agent running solidly in production before making them divide the work; reversed, you tune orchestration and infrastructure at the same time and see neither clearly.

Interview questions

  • With multiple replicas, how do you design heartbeats and health checks? Are they the same thing?多实例部署下，怎么设计心跳和健康检查？两者是同一件事吗？
    Common in ChinaCommon overseasIntermediate#observability#deployment#distributed-systems

    How to reason about it · think before answering

    1. The hinge is are they the same thing. Answering both check liveness loses the point — the interviewer wants to see you split one word into three distinct questions, because conflating them causes real outages.
    2. Separate them: a liveness probe answers should this process be restarted, a readiness probe answers can you send me traffic now, and a heartbeat dashboard answers what is the cluster's state. The audiences differ: the first two are for the orchestrator, the third is for a human.
    3. Then say why heartbeats are not optional: the orchestrator only sees process liveness, but a worker can be alive while doing no work at all — a blocked event loop, an exhausted connection pool timing out every read, a noisy neighbor saturating host CPU. This kind of zombie is exactly what the orchestrator cannot see, and only an application-level heartbeat catches it.
    4. Get the direction right too: replicas push their own heartbeat rather than the gateway polling each one. Containers change IP and hostname constantly, so a poller needs a roster that is always changing — and maintaining that roster is what heartbeats are for, so the logic is circular. Report at least three things: a timestamp for liveness, in-flight count to distinguish idle from overloaded, and a version so you can watch old and new replicas during a rollout.
    5. The sharpest point is isolation: do not query downstream dependencies inside a readiness probe. One worker going quiet would turn every gateway's readiness red, and the orchestrator would pull the entire ingress layer — turning a non-critical fault into a full outage. In reality that worker's absence does not stop intake at all: messages sit in the stream, unacked ones get claimed by someone else, and its lease changes hands when the TTL expires.
    6. Expect: so how does the gateway decide whether a worker is usable? Answer that it does not, and does not need to — the gateway never assigns work to a specific worker; the consumer group and the lease decide that. Heartbeat data is for observability and alerting, not routing. Getting here shows you actually understand the layering.

    分析过程 · 先想清楚再作答

    1. 题眼是「两者是同一件事吗」。答「都是探活」直接失分——面试官想看你能不能把一个词拆成三个不同的问题，因为混起来会造成真事故。
    2. 先拆问题：存活探针回答「这进程要不要被重启」，就绪探针回答「现在能不能给我发流量」，心跳面板回答「集群此刻是什么状态」。三者的读者不同：前两个给编排系统，第三个给人。
    3. 再说心跳为什么不可省：编排系统只能看到进程存活，而 Worker 完全可以进程活着而活儿全停——事件循环被死循环占住、连接池耗尽后取消息全超时、宿主机 CPU 被邻居打满。这类假死恰好是编排系统看不见的那种，只有业务自己上报的心跳能发现。
    4. 方向也要答对：心跳是副本自己 push，不是 Gateway 逐个 pull。因为容器随时换 IP 和主机名，去问的一方需要一份永远在变的名单，而那份名单本身就得靠心跳维护，逻辑绕回来了。上报内容至少三样：时间戳判活、在跑任务数区分闲和忙、版本号在滚动发布时看新旧两批各剩几个。
    5. 最关键的一刀是隔离性：**不要把下游依赖查进就绪探针**。一个 Worker 失联导致所有 Gateway 的就绪探针同时转红，编排系统会把整个接入层摘光——一个非核心故障被自己升级成全站不可用。而实际上那个 Worker 失联根本不影响接单：消息还在流里，没确认的会被别人接手，它的租约会因 TTL 到期而易主。
    6. 可以预期的追问：那 Gateway 怎么判断某个 Worker 可不可用？答「它不判断，也不需要判断」——Gateway 从不指定某个 Worker 干活，派活由消费组和租约决定，心跳的用途是观测和告警，不是路由。答到这里就说明你真的想清楚了分层。

    Key points

    • Split one word into three questions: liveness (restart me?), readiness (send me traffic?), heartbeat dashboard (what is the cluster doing?) — first two for the orchestrator, third for humans
    • The orchestrator sees process liveness but not zombies (blocked loop, exhausted pool, stolen CPU), so an application-level heartbeat is mandatory
    • Heartbeats must be pushed by replicas, not polled by the gateway: containers change IP constantly and polling needs a roster that heartbeats themselves maintain
    • Report timestamp, in-flight count and version — for liveness, load, and rollout progress respectively
    • Never query downstream dependencies in a readiness probe, or one quiet worker pulls the whole ingress layer and escalates a minor fault into an outage
    • The gateway does not judge worker availability — the consumer group and lease assign work; heartbeats are for observability, not routing

    答题要点

    • 一个词要拆成三个问题：存活探针（要不要重启）、就绪探针（能不能发流量）、心跳面板（集群什么状态），前两个给编排系统、第三个给人
    • 编排系统只看得见进程存活，看不见假死（事件循环卡住、连接池耗尽、CPU 被抢），所以业务层心跳不可省
    • 心跳必须是副本 push 而不是 Gateway pull：容器随时换 IP，pull 需要一份靠心跳才能维护的名单，逻辑绕回来了
    • 上报时间戳、在跑任务数、版本号三样，分别用于判活、区分忙闲、观察滚动发布进度
    • 不要把下游依赖查进就绪探针，否则一个 Worker 失联会让整个接入层被摘掉，把非核心故障升级成全站不可用
    • Gateway 不判断 Worker 可用性——派活由消费组和租约决定，心跳只用于观测告警，不用于路由
  • What is graceful shutdown, and why is killing a process outright risky? Walk through the steps.什么是优雅停机？为什么直接 kill 进程有风险？请说出具体步骤。
    Common in ChinaCommon overseasIntermediate#deployment#reliability#operations

    How to reason about it · think before answering

    1. This question tests whether you have actually shipped a release. Reciting finish in-flight work before exiting is just the definition; the interviewer wants the cost, the steps, and the ordering.
    2. Make the cost concrete. Deploys, scale-downs, host maintenance and spot reclamation all send SIGTERM, wait a grace period, then SIGKILL. SIGKILL cannot be trapped, and landing it on a worker mid-agent-loop means: the run is stuck in running forever while the user watches a spinner; you already paid for the model call but never persisted the reply; the unacked message waits for the idle threshold before anyone claims it. One deploy cuts off dozens of conversations — that is the everyday cost.
    3. Then give three steps and stress that the order is fixed. One, stop accepting work: flip a flag so the consume loop stops reading from the stream (messages already fetched but not started stay in pending for someone else, which is faster than forcing a whole batch through). Two, wait for the in-flight execution, but with a ceiling. Three, proactively release leases, deregister from the heartbeat dashboard, and exit.
    4. The ceiling in step two earns points: a hung model call means you wait forever, and the grace period will SIGKILL you anyway. Better to concede and exit — the unacked message is still pending and someone will redo it. This course uses 20 seconds, derived from the upper bound of a normal execution plus margin.
    5. Step three also earns points: leases normally change hands via TTL expiry, but that path exists for sudden death. On a planned shutdown you know you are leaving, so releasing proactively lets the successor take over on its next scan instead of waiting out a full TTL. The release must be conditional — delete only the badge that still bears your name, or you will tear down the badge of whoever just claimed it after your lease expired.
    6. Finish with two companions; miss either and the rest is wasted. The configured grace period must exceed the wait ceiling in code (code waits 20s while compose defaults to 10s, so SIGKILL lands at second 10 and your three steps only half-run). And the signal must actually reach your process (if the entrypoint is a package manager, PID 1 is the package manager, SIGTERM may never arrive, and your shutdown code never runs once).

    分析过程 · 先想清楚再作答

    1. 这题考的是「你有没有真的发过版」。答「等任务跑完再退出」只是定义，面试官要的是代价、步骤和顺序。
    2. 先把代价说具体。发版、缩容、机器维护、抢占式实例回收都会先发 SIGTERM、等宽限期、超时 SIGKILL。SIGKILL 拦不住，落到正在跑 Agent 循环的 Worker 身上：这次的 run 永远停在 running，用户界面一直转圈；模型调用的钱已经付了，回复却没落库；没确认的消息要等空闲阈值到了才被别人接手，用户白等一轮。一次发版掐断几十次对话，这就是日常代价。
    3. 然后给三步，强调顺序不能变：第一步拒新——把开关拨过去，消费循环下一轮不再从流里取消息（已经读到手上还没开始的那几条，留在 pending 里由别人接手，比硬扛完一整批更快）；第二步等手头这次执行跑完，但要有上限；第三步主动交还租约、从心跳面板注销，然后退出。
    4. 第二步的上限是加分点：一次卡死的模型调用会让你永远等不到，而宽限期一到照样 SIGKILL。与其被动挨刀，不如自己认输退出——没确认的消息还在 pending 里，别人会接手重做。本课取 20 秒，取法是「一次正常执行的耗时上限」再留余量。
    5. 第三步也是加分点：租约本来靠 TTL 到期自然易主，但那是为进程猝死准备的。计划内下线你明知道自己要走，主动交还能让接手方下一轮扫描就上岗，而不是白等一个 TTL。交还必须带条件——只删还写着自己名字的那把牌子，否则租约已过期、别人刚抢到时，你就把对方的值班牌撕了。
    6. 最后两件配套的事，漏一件前面全白做：宽限期的配置必须大于代码里的等待上限（代码等 20 秒而 compose 默认只等 10 秒，第 10 秒就 SIGKILL，三步只走到一半）；以及信号得真的传到你的进程（启动命令写成包管理器，PID 1 就是包管理器，SIGTERM 未必传得到，停机代码一次都不会执行）。

    Key points

    • Concrete cost of a hard kill: the run is stuck in running, the user stares at a spinner, the model call is paid for but the reply is unsaved, and the unacked message waits out the idle threshold
    • Three steps in a fixed order: refuse new work, wait for in-flight work with a ceiling, then release leases and deregister before exiting
    • The wait needs a ceiling (20s here): a hung model call never returns and the grace period kills you anyway, so concede — the message is still pending for someone else
    • Releasing leases proactively lets the successor start on its next scan instead of waiting a full TTL; the release must be conditional on still owning it
    • The configured grace period must exceed the in-code wait ceiling, or the three steps only half-run (stop_grace_period / terminationGracePeriodSeconds)
    • Make sure the signal reaches your process: exec the business process directly rather than letting a package manager be PID 1

    答题要点

    • 直接 kill 的具体代价：run 永远停在 running、用户界面一直转圈、模型的钱已付但回复没落库、没确认的消息要等空闲阈值才被接手
    • 三步且顺序不能变：拒绝新任务 → 等手头的跑完（有上限）→ 主动交还租约并注销心跳，然后退出
    • 等待必须有上限（本课 20 秒）：卡死的模型调用会让你永远等不到，宽限期一到照样被 SIGKILL，不如自己认输，消息还在 pending 里
    • 主动交还租约让接手方下一轮就上岗，而不是白等一个 TTL；交还必须条件化，只删还写着自己名字的那把
    • 宽限期配置必须大于代码里的等待上限，否则三步只执行到一半（compose 的 stop_grace_period / K8s 的 terminationGracePeriodSeconds）
    • 信号要真传到进程：用 exec 形式直接起业务进程，别让包管理器当 PID 1
  • During a rolling deploy, how do you keep in-flight tasks from being interrupted?滚动发布时，如何避免正在处理的任务被打断？
    Common in ChinaCommon overseasIntermediate#deployment#reliability#operations

    How to reason about it · think before answering

    1. This is the applied version of the previous question, and the difference is that it demands the orchestrator's side too — describing only the in-process steps answers half of it.
    2. The full skeleton is both sides cooperating: the orchestrator first removes traffic (turns readiness red so the load balancer stops sending new requests), then sends SIGTERM, then waits out the grace period; the process uses that window to finish in-flight work, hand back ownership, and exit cleanly. That sentence is the trunk; everything else is detail.
    3. Then distinguish the two kinds of replica, which is where the points are. A gateway has inbound connections, so draining traffic means something for it. A worker has no inbound connections at all — it pulls work from the bus, so draining for it means stop fetching new messages, which is step one of graceful shutdown. The same word is two different mechanisms on the two replica types, and saying so shows you understand pull versus push.
    4. Next, batching and ordering: replace only a subset at a time (manual batches in compose, maxUnavailable / maxSurge in Kubernetes) so enough replicas are always alive to absorb traffic. This is where the version field in the heartbeat payload pays off — you can see how many old and new replicas remain instead of deploying blind.
    5. Also mention state compatibility: during a rolling deploy old and new code run simultaneously, so schema migrations must be backward compatible (add a nullable column, dual-write, drop the old column last) and message formats cannot change in one shot. Many candidates miss this layer — however gracefully processes stop, two versions that cannot read the same data will still cause an incident.
    6. Expect: what if a single execution legitimately takes five minutes and the grace period cannot wait that long? The answer is not to stretch the grace period to five minutes but to make the task interruptible and resumable — break long work into steps that checkpoint progress (the run state machine from D11 plus at-least-once with idempotency from D9 give you exactly this), so the next replica continues the interrupted step.

    分析过程 · 先想清楚再作答

    1. 这题是上一题的应用题，区别在于它要求你把编排系统那一侧也讲进来——只讲进程内的三步只答了一半。
    2. 完整骨架是两侧配合：编排系统先摘流量（把就绪探针转红，让负载均衡不再把新请求打过来）、再发 SIGTERM、然后等宽限期；进程在这段时间里把手头的活做完、交还所有权、干净退出。这一句话就是答案的主干，剩下都是细节。
    3. 然后区分两类副本，这是拿分点。Gateway 有入站连接，摘流量对它有意义；Worker 没有任何入站连接，它是自己去总线取活的，所谓「摘流量」对它就是「自己不再取新消息」——也就是停机三步的第一步。**同一个词在两类副本上是两种机制**，能说清这一点说明你理解拉与推的差别。
    4. 接着讲批次与顺序：一次只换一部分副本（compose 里手动分批，K8s 里靠 maxUnavailable / maxSurge），保证任何时刻都有足够的存活副本接得住流量。心跳面板上的版本号字段这时派上用场——你能看到新旧两批各剩几个，而不是盲发。
    5. 还要提一句状态兼容：滚动发布期间新旧代码同时在线，所以数据库迁移必须向后兼容（先加可空列、再双写、最后才删旧列），消息格式也不能一次性改。这是很多人漏掉的一层——进程停得再优雅，新旧版本读不了同一份数据照样出事故。
    6. 可以预期的追问：如果一次执行本来就要跑 5 分钟，宽限期不可能等那么久怎么办？答案不是把宽限期拉到 5 分钟，而是让任务可中断可重入——把长任务切成可保存进度的小步（D11 的 run 状态机和 D9 的 at-least-once 加幂等正好提供了这个基础），被打断的那一步由下一个副本接着做。

    Key points

    • The full skeleton is both sides: orchestrator drains traffic, sends SIGTERM, waits the grace period; the process finishes in-flight work, hands back ownership, exits cleanly
    • Draining means two different things for gateways and workers: readiness turning red versus the worker itself stopping its fetch from the bus
    • Replace in batches (maxUnavailable / maxSurge or manual) so enough replicas stay alive; the version field in heartbeats shows how many old and new remain
    • Old and new code run concurrently, so migrations must be backward compatible (nullable column, dual-write, drop last) and message formats cannot change in one step
    • Long tasks are not solved by a longer grace period but by being interruptible and resumable — checkpointed steps that the next replica can continue

    答题要点

    • 完整骨架是两侧配合：编排系统先摘流量、再发 SIGTERM、等宽限期；进程在这段时间做完手头的活、交还所有权、干净退出
    • Gateway 和 Worker 的「摘流量」是两种机制：前者靠就绪探针转红让负载均衡停止转发，后者靠自己不再从总线取新消息
    • 分批替换（maxUnavailable / maxSurge 或手动分批），保证任何时刻有足够存活副本；心跳里的版本号让你看到新旧两批各剩几个
    • 新旧代码同时在线，所以数据库迁移必须向后兼容（加可空列 → 双写 → 最后删旧列），消息格式不能一次性改
    • 长任务不该靠拉长宽限期解决，而要做成可中断可重入：切成能保存进度的小步，被打断的那步由下一个副本接着做
  • How do you isolate dev from prod so local development cannot touch production data?怎么设计 dev 与 prod 的隔离，防止本地开发影响线上数据？
    Common in ChinaCommon overseasBasic#operations#security#configuration

    How to reason about it · think before answering

    1. This looks basic, but it screens for whether you have been burned. People who have start with the failure shape; people who have not start with use different config files.
    2. Describe the failure: same codebase, often the same Redis, and you start a worker locally to debug — except it is connected to the production stream and it claims and executes a real user's message. There is no error anywhere and both sides log business as usual, because from the code's point of view it did dutifully process one message. Precisely because nothing errors, this can run for a long time before anyone notices.
    3. Then give layered options by cost: namespacing (shared infrastructure, prefixed keys), separate instances (its own Redis and database), and separate environments (network, credentials, accounts all split). Production eventually wants the third layer, but the first is the cheapest and the easiest to get wrong, so that is where the focus belongs.
    4. The implementation detail in layer one is where the points are: the prefix may only be assembled in one function. Scatter string concatenation around the codebase, miss one key out of twenty, and you have no isolation at all — and the one you missed is usually the newest, least tested feature. This point signals real experience more than add a prefix does.
    5. Add three companions. Split credentials, so the local key can only reach the dev database and a misconfiguration cannot reach production. Make destructive operations environment-aware: scripts that truncate tables, replay dead letters or rebuild indexes read the environment variable on their first line and demand explicit confirmation in production. And forbid fallback implementations in production: if a config slip makes production take the in-memory path, processes come up quietly, each working in its own memory, with every health check green — that kind of fault hides for hours, so failing fast at startup is far cheaper than diagnosing it later.
    6. Expect: why not just use separate instances and skip prefixes? Because separate instances solve connected to the wrong address while prefixes solve connected to the right address but the wrong namespace — the two fail differently. Prefixes are also nearly free, and they incidentally isolate each developer's data in a shared test environment. Defense should be layered, and there is no reason to skip the cheapest layer.

    分析过程 · 先想清楚再作答

    1. 这题看着基础，但它筛的是「有没有踩过」。踩过的人第一句会说事故形态，没踩过的人第一句说「用不同的配置文件」。
    2. 先说事故形态：同一套代码、经常还是同一个 Redis，你在本机起一个 Worker 调试，它连的却是线上那条流，把真实用户的消息捞走执行了。**这类事故没有任何报错，两边日志都显示一切正常**——从代码角度看它确实老老实实处理了一条消息。正因为没有报错，它可能持续很久才被发现。
    3. 然后按成本分层给方案：命名空间（同一套基础设施，键名带前缀）、独立实例（各自的 Redis 与数据库）、独立环境（网络、凭证、账号全分开）。生产系统最终要走到第三层，但第一层成本最低也最容易漏，所以是重点。
    4. 第一层的关键实现细节是拿分点：前缀只能在一个函数里拼。散落到各处去拼字符串，二十个键名里漏掉一个就等于没隔离，而漏掉的那个通常是最新加、最没被测过的功能。这一点比「要加前缀」本身更能体现工程经验。
    5. 再补三件必须一起做的事：凭证分开（本机那把 key 只能连开发库，配置写错也波及不到线上）；破坏性操作要认环境（清库、重放死信、重算索引这类脚本第一行先读环境变量，生产上要求显式确认）；生产禁止降级实现（离线用的内存实现在生产上一旦因配置疏漏被走到，进程会安静起来、各自在自己内存里干活，健康检查还全是绿的，这类故障能藏好几个小时——启动时直接报错退出比事后排查便宜得多）。
    6. 可以预期的追问：为什么不干脆只用独立实例，省掉前缀这一层？答：独立实例解决的是「连错了地址」，前缀解决的是「连对了地址但走错了命名空间」——两者失效的方式不同。而且前缀几乎零成本，在共享测试环境、多人并行开发时还能顺带隔离每个人的数据。防御要分层，最便宜那层没理由不做。

    Key points

    • Lead with the failure shape: a local worker attached to the production stream claims and runs a real user's message, with normal logs on both sides and no error, so it hides for a long time
    • Three layers by cost: namespacing (key prefixes), separate instances (own Redis and DB), separate environments (network, credentials, accounts)
    • The prefix must be assembled in exactly one function — scattered concatenation misses one key and voids the isolation, usually the newest and least tested feature
    • Split credentials so the local key only reaches dev; destructive scripts read the environment first and require explicit confirmation in production
    • Forbid the in-memory fallback in production: on a config slip processes come up quietly with green health checks and the fault hides for hours — fail fast at startup instead
    • Separate instances prevent wrong address, prefixes prevent right address wrong namespace — different failure modes, and the cheapest layer is free

    答题要点

    • 先说事故形态：本机 Worker 连上线上流，把真实用户消息捞走执行，且两边日志都显示正常、没有任何报错，所以能藏很久
    • 按成本分三层：命名空间（键名前缀）、独立实例（各自 Redis 与库）、独立环境（网络凭证账号全分开）
    • 前缀只能在一个函数里拼——散落各处漏掉一个键就等于没隔离，而漏掉的通常是最新加、最没测过的功能
    • 凭证分开，本机 key 只能连开发库；破坏性脚本第一行读环境变量并在生产要求显式确认
    • 生产禁止降级到内存实现：配置疏漏时进程会安静起来、健康检查全绿，故障能藏几小时，应在启动时直接报错退出
    • 独立实例防「连错地址」、前缀防「地址对了但命名空间错了」，失效方式不同，最便宜那层没理由不做
  • System design: design an IM agent platform where users chat with an AI assistant inside a messaging app. The assistant calls tools, remembers long-term preferences, and proactively pushes scheduled messages. Target 100k daily active users.系统设计：请设计一个 IM Agent 平台——用户在即时通讯软件里和一个 AI 助手对话，助手能调用工具、记住长期偏好、还能定时主动推送。要求支撑十万日活。
    Common in ChinaCommon overseasDeep dive#system-design#distributed-systems#cost#operations

    How to reason about it · think before answering

    1. Do not start drawing. The most common way to fail a design question is to hear the prompt and immediately sketch boxes, only for the interviewer to realize twenty minutes later that you solved a different problem. Spend three to five minutes on four questions: traffic shape (how many concurrent sessions does 100k DAU imply, and what is the peak-to-trough ratio), latency (how fast must first byte be, is streaming required), the nature of the tools (read-only lookups, or writes with side effects), and the compliance boundary on proactive pushes (may you push at night, what is the daily cap). All four change the architecture materially, so asking them is itself worth points.
    2. Then state the trunk in one sentence: stateless ingress, a message bus for decoupling, stateful workers sharded by user, all state in the database. Walk the data flow: the messaging platform's webhook hits ingress, which does only auth, rate limiting, persistence and publish, and returns 202 immediately; the execution side pulls work, runs the agent loop, and streams output fragments back; proactive pushes come from a central scheduler publishing onto the same bus. The load-bearing argument is that ingress latency is bounded while execution latency is not, so putting them in one process means one slow model call occupies a connection that should have returned in milliseconds — say this out loud, it is the premise of the whole answer.
    3. Then justify each module. Storage: sessions, runs and messages, with runs existing separately because only it can answer whether this attempt actually finished; idempotency comes from a unique constraint on runs, not from check-then-insert. Bus: Redis Streams consumer groups for fan-out, at-least-once semantics, with exactly-once manufactured by consumer-side idempotency, and messages that fail three times moved to a dead-letter stream. Ordering: the consumer group's unit of assignment is one message while the business requires serialization per user, so hash userId into a fixed set of shards and let exactly one worker hold each shard's lease. Memory: embeddings in pgvector, retrieval wrapped as a tool the model chooses to call, with no identity parameter — identity only ever comes from the session.
    4. Treat proactive push as its own section, because it is what separates this from an ordinary chat service. The central scheduler publishes one message on a time match and the execution side is unchanged; the idempotency key is anchored to the scheduled minute, so replaying after a scheduler restart cannot double-send. For compliance you need timezone, quiet hours and a daily cap — and all three must be evaluated before publishing rather than at send time, or you have already paid for the model call before discovering you should not have pushed.
    5. Then volunteer capacity and cost numbers, which is what separates senior candidates. 100k DAU at ten turns each is a million model calls; at roughly a thousand tokens in and out, with input at $0.15 and output at $0.60 per million tokens, that is about $750 a day. That number immediately implies three requirements: meter token usage per call and convert to dollars (otherwise you cannot tell which user or feature is burning money), build tiered degradation (push over-budget users to a cheaper model rather than refusing them), and recognize that context length is the dominant cost lever (so compress history and cap retrieved items).
    6. Land on operability, which is this week's payoff: multiple replicas, heartbeats to surface zombies, readiness probes that only check their own hard dependencies, graceful shutdown so deploys do not cut conversations, and dev/prod isolation via key prefixes. Pair every mechanism with what happens when it fails — leases can split-brain so you need a self-fencing rule and fencing tokens, heartbeats produce false positives so a red dashboard alerts a human rather than auto-draining, shutdown can time out so the wait needs a ceiling. A mechanism without a stated failure mode reads as something you only read about.
    7. Expect, in rough order of frequency: where are the single points (the scheduler is stateless and restartable; Redis and Postgres rely on managed primary/replica); how do you roll out safely (old and new workers coexist and a version field in the message selects the prompt set); what if the user sends another message mid-reply (merge a change of mind within thirty seconds into the same execution rather than running two concurrently); and how would you halve the cost (cache frequent answers, compress history, route simple intents to a smaller model).

    分析过程 · 先想清楚再作答

    1. 先别画图。系统设计题最常见的死法是听完就开始画框，二十分钟后面试官发现你解的是另一道题。花三到五分钟问清四件事：一是流量形状（十万日活对应多少并发会话、峰谷比多少），二是延迟要求（首字节要多快，是否必须流式），三是工具的性质（只读查询还是有写操作和副作用），四是主动推送的合规边界（能不能在深夜推、每天上限几条）。这四个答案会实质改变架构，问它们本身就是分数。
    2. 然后给主干，一句话先定形状：**接入层无状态、消息总线解耦、Worker 有状态且按用户分片、状态全在数据库**。接着按数据流走一遍：IM 平台的 webhook 打到接入层，接入层只做鉴权、限流、落库、投递四件事，立刻返回 202；执行侧从总线取活、跑 Agent 循环、把输出片段回传；主动推送由一个中心调度器按时间投递进同一条总线。**关键论点是接入层耗时确定、执行层耗时不确定，把它们放在一个进程里意味着一次慢的模型调用会占住一个本该毫秒级返回的连接**——这是整道题的立论基础，要主动说出来。
    3. 再逐个模块给出选择和理由。存储：sessions / runs / messages 三张表，runs 单独存在是因为只有它能回答「这次到底跑完没有」，幂等靠 runs 上的唯一约束而不是先查后插。总线：Redis Streams 的消费组做分摊，语义是至少一次，恰好一次靠消费端幂等做出来；反复失败的消息投递三次后进死信流。顺序：消费组的分配单位是一条消息而业务要求的串行单位是一个用户，所以按 userId 哈希到固定数量分片，每个分片同一时刻只有一个 Worker 持有租约。记忆：pgvector 存 embedding，检索包成一个工具交给模型自己决定要不要查，且不给它身份参数——身份只能来自会话。
    4. 主动推送这一块要单独讲透，因为它是这道题区别于普通聊天服务的地方。中心调度器命中时间点后只投一条消息，执行侧照旧；幂等键锚在「计划触发的那一分钟」，所以调度器崩溃重启后回看重放不会重复推送。合规上要有时区、静默时段、每日上限三道闸，而且这三道闸必须在投递前判断而不是在推送时判断——否则你已经花了模型调用的钱才发现不该推。
    5. 然后主动给出容量和成本的数字感，这是高级候选人的分水岭。十万日活、人均十轮对话是一百万次模型调用；按输入输出各一千 token、每百万 token 输入 0.15 美元输出 0.60 美元估算，一天大约七百五十美元。这个数字立刻推出三件事必须做：token 用量要按调用记账并换算成美元（否则你无法定位是哪个用户或哪个功能在烧钱）、要有分层降级（超预算的用户切便宜模型而不是直接拒绝）、以及上下文长度是主要成本杠杆（所以要压缩历史、控制检索条数）。
    6. 最后收在可运维性上，也就是这一周的落点：多副本部署、心跳发现假死、就绪探针只查自己必需的依赖、优雅停机让发版不掐断对话、dev 与 prod 用键名前缀隔离。**每个机制都要配一句「它失效时会怎样」**——租约会脑裂所以要有自杀规则和护栏令牌、心跳会误判所以面板转红只告警不自动摘流量、停机会超时所以等待要有上限。说不出失效模式的机制，面试官会认为你只是读过。
    7. 可以预期的追问，按出现频率排：单点在哪（调度器无状态可重启，Redis 和 Postgres 靠托管服务的主备）；怎么灰度（新旧 Worker 同时在线，靠消息里的版本字段决定走哪套提示词）；用户在助手回复中途又发一句怎么办（三十秒内的改口合并进同一次执行，而不是并发开两个）；成本再降一半怎么做（缓存高频问答、压缩历史、把简单意图路由到小模型）。

    Key points

    • Spend three to five minutes clarifying four things: traffic shape, latency targets, whether tools have side effects, and the compliance boundary on proactive pushes
    • State the trunk in one sentence: stateless ingress, bus for decoupling, stateful workers sharded by user, all state in the database — premised on bounded ingress latency versus unbounded execution latency
    • Storage is sessions/runs/messages with idempotency from a unique constraint on runs; the bus is Redis Streams consumer groups, at-least-once plus consumer idempotency, dead-lettering after three failures
    • Ordering comes from hashing userId into shards plus leases: the consumer group assigns per message while the business serializes per user
    • Memory is pgvector exposed as a tool the model may call, with no identity parameter — identity comes only from the session
    • Proactive push flows through a central scheduler with the idempotency key anchored to the scheduled minute; timezone, quiet hours and daily caps are enforced before publishing
    • Bring numbers: 100k DAU at ten turns is ~1M calls and ~$750/day, which implies metering, tiered degradation, and context length as the main cost lever
    • Land on operability: replicas, heartbeats for zombies, readiness probes scoped to own dependencies, graceful shutdown, dev/prod prefix isolation
    • Pair each mechanism with its failure mode — leases split-brain, heartbeats false-positive, shutdown times out; a mechanism without one reads as book knowledge

    答题要点

    • 先用三到五分钟问清四件事：流量形状、延迟要求、工具是否有副作用、主动推送的合规边界——它们会实质改变架构
    • 主干一句话：接入层无状态、消息总线解耦、Worker 有状态且按用户分片、状态全在数据库；立论是接入层耗时确定而执行层不确定
    • 存储 sessions / runs / messages 三张表，幂等靠 runs 上的唯一约束；总线用 Redis Streams 消费组，至少一次加消费端幂等，三次失败进死信
    • 顺序靠 userId 哈希分片加租约：消费组的分配单位是一条消息，而业务要求的串行单位是一个用户
    • 记忆用 pgvector 并包成工具交给模型自己决定是否检索，不给身份参数——身份只能来自会话
    • 主动推送由中心调度器投递，幂等键锚在计划触发的那一分钟；时区、静默时段、每日上限三道闸必须在投递前判断
    • 给出成本数字感：十万日活人均十轮约一百万次调用、一天约七百五十美元，由此推出计量记账、分层降级、压上下文三件事
    • 收在可运维性：多副本、心跳查假死、就绪探针只查自己的依赖、优雅停机、dev/prod 前缀隔离
    • 每个机制都配一句失效模式：租约会脑裂、心跳会误判、停机会超时——说不出失效模式等于只是读过

Comments