Dayward AI
中文
Week 2 · D9About 6 hours

A Redis Streams Message Bus: XADD/XREADGROUP/XACK/XAUTOCLAIM, Consumer Groups, Poison Messages

Build a message bus with Redis Streams, understand how a consumer group lets multiple workers split consumption, and handle a "poison message" that keeps failing.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Write a message with XADD, consume it as a consumer group with XREADGROUP, and XACK it
  2. Explain what problem each of XACK and XAUTOCLAIM solves
  3. Design a simple isolation mechanism for a poison message (one that keeps failing)

Yesterday split the service into a gateway, a message bus, and a worker, and that middle bus is still just a box on the diagram: POST /chat writes a run into the runs table and returns, and nobody answers who executes it afterwards. Today we fill in that box — and by the end of this chapter the full D8-to-D9 chain genuinely runs for the first time.

Plain-Language Walkthrough

Once it is persisted, who does the work

Think of the parcel pickup point by the entrance to a residential block. The van arrives and the driver does not carry each parcel door to door waiting for you to answer. They shelve it, record a shelf number, and leave; collection is somebody else's job. Shelving and delivery are completely separated by that shelf: the van can keep unloading, and one parcel nobody ever collects does not block anyone behind it.

At the end of D8 your service was stuck in exactly the state of having no shelf. POST /chat authenticates, rate-limits, writes to the database, returns a runId, and stops. The most immediate idea is: the request is already in hand, so just execute it here and return when done. D8 priced that road — one slow execution stalls the whole process's event loop — and here are three harder reasons.

One, the traffic shapes do not match. Users arrive in bursts, and ten minutes of morning peak may equal an hour of the flat period. The intake layer must be provisioned for peak; the execution layer only needs the average, letting backlog queue on the shelf. Cram them into one process and you are forced to provision the execution layer for peak too, paying for processes that idle most of the day.

Two, the failure boundaries do not match. Model wobbles, tool timeouts, one execution running the full 40 seconds — all routine for the execution layer. Share a process with accepting requests and one execution's failure reaches connections mid-handshake, so the user sees "the site is down" when in truth some model was slow.

Three, the scaling units do not match. The intake layer runs out of connections and memory, the execution layer runs out of concurrency and egress bandwidth, and combined you can only add both at once.

Could you skip Redis and use the runs table as the queue? select ... for update skip locked genuinely works, and a dozen messages a second is entirely fine. Its problem is polling: set the interval to 1 second and the user waits up to a second for nothing; set it to 50 milliseconds and an idle database gains 20 pointless queries a second multiplied by the worker count. A message bus's value is push — XREADGROUP's BLOCK parameter lets a worker hang there and wake the instant a message exists.

State the price too: one more component to operate, one more hop of latency, and messages will definitely be delivered more than once — which is section four's subject.

In code today's change is two lines: after persisting, publish the id.

gateway.js
// Dependencies: ioredis 5.x. createRun is D8's insert, and inserted tells you whether
// a row was really added
const { runId, inserted } = await store.createRun({ sessionId, idempotencyKey, input: text })
 
// Hit the idempotency_key unique constraint: this is a client retry, somebody is already
// running it, so do not publish again
if (!inserted) return reply.send({ runId, duplicated: true })
 
// The message body carries only ids, not the text: a bus is routing, not a database
const streamId = await redis.xadd('koda:runs', '*', 'runId', runId, 'sessionId', sessionId)
reply.send({ runId, streamId })

Note the order of those two steps: persist first, publish second. Reversed, a worker may well take the message before that runs row has committed and read a runId that does not exist.

Even in the right order, one hole remains: the write succeeds and XADD fails, so that run sits at pending forever with nobody executing it and nobody raising an error. The cheap fix is a periodic job that scans for runs created more than 30 seconds ago and still pending, and republishes them. As long as the bus and the database are two systems, this hole necessarily exists; you can compensate for it, not eliminate it. Volunteering that in an interview is worth more than any amount of fluency about XADD.

So how is the shelf actually used? Redis Streams has only four commands to remember, and together they happen to constitute a complete delivery semantics.

Four commands, one delivery semantics

Back to the pickup point. A parcel's full life on the shelf is four actions: shelved, taken by a courier and logged onto the out-for-delivery list, signed for, so the record is cleared, and the courier had an accident and never returned, so somebody else takes over their parcels. Redis Streams' four commands map one to one:

Pickup-point actionCommandWhat it does
shelve the parcelXADDappend a message to the stream's tail, returning an increasing id
take it and log it out for deliveryXREADGROUPassign the message to a consumer and record it on the pending list
sign for it, clear the recordXACKremove the message from the pending list
take over somebody's overdue parcelsXAUTOCLAIMreassign pending messages idle too long to another consumer

Walking through it in redis-cli is clearer than ten paragraphs:

TextText
127.0.0.1:6509> XGROUP CREATE koda:runs workers 0 MKSTREAM
OK
127.0.0.1:6509> XADD koda:runs * runId run-1 sessionId s1
"1757000000123-0"
127.0.0.1:6509> XREADGROUP GROUP workers worker-a COUNT 10 STREAMS koda:runs >
1) 1) "koda:runs"
   2) 1) 1) "1757000000123-0"
         2) 1) "runId"  2) "run-1"  3) "sessionId"  4) "s1"
127.0.0.1:6509> XPENDING koda:runs workers
1) (integer) 1          # 1 delivered and unacknowledged
2) "1757000000123-0"
3) "1757000000123-0"
4) 1) 1) "worker-a"  2) "1"
127.0.0.1:6509> XACK koda:runs workers 1757000000123-0
(integer) 1             # 1 record really cleared
127.0.0.1:6509> XACK koda:runs workers 1757000000123-0
(integer) 0             # acking again returns 0: XACK is idempotent by construction

Three details are worth memorizing. One, a message id has the shape of a millisecond timestamp plus a sequence number within that millisecond, so it is monotonically increasing by nature and replaying every message after a given moment is a free capability. Two, the greater-than sign at the end of XREADGROUP is part of the command, meaning "give me new messages never delivered to anyone in this group"; replace it with 0 and it means "re-read the ones on my own pending list," so new messages never arrive and no error is raised. Three, XACK returns how many records were genuinely cleared, and acking the same one twice returns 0, so a redundant ack is always safe.

There is one engineering cost, and it is a hard one: a stream only grows. XACK deletes the record on the pending list; the message itself stays in the stream. At roughly 100 bytes per message, a million a day is 100MB and a month is 3GB — and Redis is an in-memory database, so that arithmetic is mandatory. The fix is MAXLEN on XADD, or a periodic XTRIM.

One stream, three workers sharing it

Three couriers stand at the same shelf. Joined to the same crew, each parcel is taken by exactly one of them and their throughput adds up; each keeping a private logbook with no coordination, the same parcel gets delivered three times. The former is a work queue, the latter is publish-subscribe.

Redis Streams supports both through one concept, the consumer group: multiple consumers within one group split the messages, while different groups each receive everything. This course uses one group, always named workers.

TextText
                    +---------------- consumer group: workers -----------------+
                    |                                                          |
koda:runs stream    |   worker-1 --+                                           |
  [m1][m2][m3][m4] -+-) worker-2 --+-) each message reaches exactly one worker  |
  (grows only)      |   worker-3 --+   delivered-but-unacked ones sit in pending|
                    +----------------------------------------------------------+

That pending list is the hub of the whole mechanism; Redis calls it the PEL (pending entries list). It records three things: whose message this is, how many times it has been delivered, and when it was last delivered. The delivery-count column matters: section five's poison-message decision reads it directly, with no separate counter table.

The worker's consumer loop therefore always has the same shape: first pick up what someone else dropped, then take new work.

worker.js
// The order cannot be reversed: XAUTOCLAIM before XREADGROUP. The other way round, an
// abandoned message queues forever behind new ones and never gets its turn when busy
const [, claimed] = await redis.xautoclaim('koda:runs', 'workers', me, 30000, '0', 'COUNT', 10)
const fresh = claimed.length
  ? []
  : await redis.xreadgroup('GROUP', 'workers', me, 'COUNT', 10, 'BLOCK', 5000, 'STREAMS', 'koda:runs', '>')
 
for (const msg of normalize(claimed, fresh)) {
  try {
    await execute(msg.fields.runId)
    // XACK has to be the last step. Clearing the record early means one crash
    // vaporizes the message
    await redis.xack('koda:runs', 'workers', msg.id)
  } catch (err) {
    // Do nothing: the message stays in pending, waiting for XAUTOCLAIM to hand it
    // to the next idle worker
    log.warn({ err, id: msg.id }, 'processing failed, left in pending for redelivery')
  }
}

There is an easily overlooked choice here: the consumer's name. If a container takes a random name on every restart, the unacked messages under the previous name become orphans nobody claims, and only XAUTOCLAIM can recover them. So either make the name stable (using the ordinal a stateful deployment gives you) or rely on XAUTOCLAIM as the backstop and periodically XGROUP DELCONSUMER the dead names. This course takes the latter.

XAUTOCLAIM's idle threshold needs equal thought. Thirty seconds works because one execution's normal duration is far below it; give it 100 milliseconds and a message still being processed normally gets snatched away, so the same run executes twice. But raising the threshold only lowers the probability, it does not eliminate duplication — the real backstop is the next section's two gates.

At-least-once: you will definitely receive the same message twice

The worker finishes and, in the instant before XACK, the process is killed. The business work is done and that record is still on the pending list. Once the idle threshold elapses, XAUTOCLAIM hands it to another worker, and the same work is done a second time.

That window cannot be closed: writing the business data (Postgres) and clearing the record (Redis) are two writes in two systems, and unless they sit in one transaction there is always a gap where a crash can happen. So the three semantics rank like this:

  • at-most-once: XACK first, then work. The price is losing the item outright on a crash — do not use it outside "losing one is fine" cases such as analytics events.
  • at-least-once: work first, then XACK. At least once and possibly more. This is what Redis Streams, Kafka, and SQS all provide, and it is this course's choice.
  • exactly-once: requires merging the two writes into one atomic commit. Kafka's transactions achieve it inside a read-Kafka-write-Kafka loop, and the moment the downstream is a database or a third-party API you are back to at-least-once.

So the correct thing to say in an interview is: exactly-once is not something the bus gives you, it is an effect the consumer's idempotency produces. Saying only "the business has to be idempotent" is empty; you have to point at a specific constraint and say where it blocks. This course's two gates both live on the tables finalized on D8:

SQLSQL
-- Gate one: the client retries POST /chat. However many times one intention is retried,
-- it produces one run
insert into runs (id, session_id, status, idempotency_key, input)
values ($1, $2, 'pending', $3, $4)
on conflict (idempotency_key) do nothing
returning id;
-- Zero rows returned = the unique constraint was hit = this is a retry, so the gateway
-- returns the existing runId and does not XADD again
 
-- Gate two: the same bus message is delivered twice. One run's one sequence number can
-- hold only one reply
insert into messages (id, session_id, run_id, role, content, seq)
values ($1, $2, $3, 'assistant', $4, 1)
on conflict (run_id, seq) do nothing;
-- Even if two workers genuinely finish the same run at once, the user sees one reply

The first gate blocks duplicate submission and the second blocks duplicate execution. Between them you can add a cheap short circuit: after taking a message, the worker reads runs once and, finding the status already done, just XACKs and leaves — that step saves the money of one model call, which is an optimization. Correctness itself rests on those two unique constraints.

Poison messages: one bad message can drag down a whole stream

Some message fails no matter who processes it: the input carries an order number that has been deleted, or a field whose format was wrong when it was written. By the previous section's rule, a failure means no XACK and staying in pending for redelivery, so it enters an endless cycle: delivered, failed, idle-timed-out, claimed, failed again. It will never improve and it consumes worker capacity throughout. One bad message dragging down a whole stream is what a poison message is.

The pickup point's answer is unglamorous: a parcel whose address can never be reached must not be attempted daily; it moves to the problem-parcel area for somebody to handle.

This course fixes the rule as three numbers:

  1. The criterion is the delivery count the PEL records itself, with no separate counter table.
  2. The threshold is 3: still failing after 3 deliveries means isolation.
  3. The isolation action is move it to the koda:runs:dead dead-letter stream, then XACK the original stream, while marking the run failed with the reason recorded.

All three actions are required. Move without XACK and it still sits in pending waiting to be claimed; XACK without moving and the message vanishes along with the reason for the failure, leaving the user on "thinking" forever.

poison.js
const POISON_THRESHOLD = 3
 
async function onFailure(msg, err) {
  // Below the threshold: do nothing. The message stays in pending for XAUTOCLAIM to hand
  // to the next worker
  if (msg.deliveryCount < POISON_THRESHOLD) return
 
  // At the threshold: move, clear, mark failed - all three are required
  await redis.xadd('koda:runs:dead', '*',
    'runId', msg.fields.runId, 'originalId', msg.id,
    'deliveries', String(msg.deliveryCount), 'reason', err.message)
  await redis.xack('koda:runs', 'workers', msg.id)
  await store.markFailed(msg.fields.runId, err.message)
}

The full trajectory of the boom message in today's lab looks like this:

TextText
[worker-c] run=run-6 failed for the 1st time, left in pending for redelivery
[worker-c] XAUTOCLAIM took over 1757000000475-1 (delivery 2)
[worker-c] run=run-6 failed for the 2nd time, left in pending for redelivery
[worker-c] XAUTOCLAIM took over 1757000000475-1 (delivery 3)
[worker-c] run=run-6 still failing after 3 deliveries, moved to koda:runs:dead
[5/5] poison isolation: OK - 3 failures, 1 in the dead-letter stream, run.status=failed, pending 0

Setting the threshold at 3 is a trade-off: at 1, one network wobble condemns a message that would have succeeded; at 10, you waste ten executions' money and time on a message that was always going to die. Also note there is no exponential backoff here: Redis Streams' redelivery timing is decided by the idle threshold, and backoff would mean XADDing the message again with a next-eligible time, which is implementing a delay queue. This course does not, but you should know it is one gap Streams has next to a purpose-built message broker.

Redis Streams or Kafka

This question comes up astonishingly often, and the right answer is not "it depends on data volume."

DimensionRedis StreamsKafka
Where data livesmostly memory, persisted via AOF/RDBmostly disk, sequential writes, built for long retention
Retention policyyour own MAXLEN / XTRIM trimmingconfigured by time or size, weeks is common
Ordering guaranteeordered within one stream; assignment inside a group is arbitraryordered within a partition, same key to the same partition
Consumer scalingadd as many consumers to a group as you likebounded by partition count; consumers beyond that idle
Replayre-read from any point by message idoffset reset, considerably more mature
Operational costyou probably already run Redis, so nothing newat least a cluster plus coordination, needs an owner

The test is these two sentences: how long the messages need to be kept, and whether a second class of consumer will appear. If the lifecycle is "useless once executed" and the worker is the only consumer, Streams is entirely sufficient and saves you a whole operational surface. If you need replay from any point in the last three months, or the same data has to feed real-time execution, an offline warehouse, and risk control simultaneously, then Kafka.

And one thing that must be said honestly: Redis persistence is lossy. AOF flushes once a second by default, so the worst case loses the last second of writes; replication is asynchronous, so unsynchronized messages disappear on a failover. Therefore in this architecture the source of truth is always the runs table in Postgres and the stream is only a trigger — lose a message and that run is still pending, and the republish job picks it back up. Treating the bus as the sole data store is the most dangerous misuse of this architecture.

One deployment note in passing: stream names need a dev: or prod: prefix (taken from APP_ENV), or messages from local debugging get taken and executed by production workers. D14 uses this detail properly.

The chain now works, and the consumer group has buried a landmine: which worker a message goes to depends entirely on who asks first. For today's echo that is fine; for a user sending two sentences in a row it is a disaster — two workers process them concurrently, whoever writes the messages table first comes first, and the user sees the conversation out of order. That is tomorrow's problem, head on.

Source Reading

Hands-On Lab

🧪 D9 lab: gateway publishes, worker consumes and echoes

Code location: labs/agent-30days/day-09-redis-streams-bus

Acceptance criteria:

  1. All five self-checks under MOCK=1 SELFTEST=1 pnpm start pass. starter/ as-is passes 1 and 2 and fails 3, 4 and 5, and only completing the four exercise points turns it all green.
  2. Check 2 shows both workers processing messages (in the form a=2 b=1), 3 replies, and pending 0: the consumer group really is splitting the load and pending drops to zero after XACK.
  3. Check 3 shows the duplicate submission being blocked, 3 runs, and 3 replies: at-least-once delivery did not become two replies.
  4. Check 4 shows pending 1 after the crash and worker-b taking over 1: XAUTOCLAIM really let another worker take over and finish.
  5. Check 5 shows 3 failures, 1 entry in the dead-letter stream, run.status=failed, and pending 0; pnpm typecheck passes with no any.

Under MOCK=1 you need no Redis, no Docker, and no API key: src/infra/memory-bus.ts is an in-memory implementation rather than a stub, with the read cursor, the pending list, and the delivery counts genuinely written out, so the behavior offline matches real Redis exactly. To verify that, docker compose up -d a redis:7-alpine, swap MOCK=1 for REDIS_URL, and run again — same business code, same five green checks. The business processing is deliberately only an echo today; replace generateReply in worker.ts with D4's model-calling layer and not one line of the rest of the chain changes. If you get stuck, read the README's common-traps section first.

  1. Run MOCK=1 SELFTEST=1 pnpm start against starter/ first and remember the 2-of-5 figure and the wording of the three failures.
  2. Exercise 1: make createRun in shared/store.ts do on conflict do nothing on the idempotency key so check 3 turns green — you will watch the run total fall from 4 back to 3.
  3. Exercises 2 and 3: in infra/memory-bus.ts, record each delivery onto the pending list (consumer, delivery count, delivery time), then implement autoClaim to reassign by idle duration and increment the delivery count. Check 4 turns green and the takeover line appears in the log.
  4. Exercise 4: in worker/worker.ts add the dead-letter move — at a delivery count of 3, publish to koda:runs:dead, XACK the original stream, and mark the run failed. Check 5 turns green.
  5. Bring up a real Redis, run the self-checks again, and use redis-cli's XLEN, XINFO GROUPS, and XPENDING to reconcile the three numbers the self-check printed.

Interview Questions

Today's five questions are in the bank below, weighted toward the boundary between at-least-once and exactly-once, the Streams-versus-Kafka decision, and dead-letter handling. Expand a question and read the analysis before the key points — question 3 on idempotency is where this chapter gets pressed hardest, so do not skip it. Each is tagged for the China-domestic or overseas market so you can prioritize by where you are applying.

Checklist and Tomorrow

  • Write a message with XADD, consume it as a consumer group with XREADGROUP, and XACK it
  • Explain what problem each of XACK and XAUTOCLAIM solves
  • Design a simple isolation mechanism for a poison message (one that keeps failing)
  • Name the three things a consumer group's pending list records, and why the poison decision needs no separate counter table
  • Point at the two unique constraints and say where each blocks at-least-once delivery, rather than only saying "the business has to be idempotent"
  • All 5 acceptance criteria of the lab pass, including a run against real Redis
  • Answer at least 4 of the 5 interview questions without looking at the key points

Tomorrow (D10) we defuse the landmine today left behind: a consumer group hands a message to any idle worker, so one user's two sentences are processed by two workers at once and the replies come back out of order. The approach is hashing users by id onto 256 fixed shards, with each shard held by exactly one worker at a time via a lease. Why the bus before sharding? Because without messages moving you cannot see the out-of-order symptom at all; and if sharding came first you would assume it is a lock for Redis, when it is really there to preserve one user's conversation order.

Interview questions

  • How does a Redis Streams consumer group work, and why can it serve both as a work queue and as pub/sub?Redis Streams 的 consumer group 是怎么工作的？为什么它既能做工作队列又能做发布订阅？
    Common in ChinaCommon overseasBasic#message-bus#redis-streams

    How to reason about it · think before answering

    1. This is a concept question; the discriminator is whether you separate the group layer from the consumer layer. Saying only 'several consumers read together' invites 'so is a message processed twice?' — and that is exactly what the two layers settle.
    2. Give the structure: the stream is append-only; a group sits on the stream and owns a read cursor plus a pending list; a consumer is just a name inside a group. Consumers in one group share the messages (each message goes to exactly one of them), while separate groups each see the full stream — one data structure, both a work queue and pub/sub.
    3. Then name the three things the pending entries list records: which consumer owns the message, how many times it has been delivered, and when it was last delivered. Those map to 'who is working on it', 'is it poison yet' and 'can someone else take over' — knowing them signals you read the docs, not just a snippet.
    4. Land on the dispatch rule: a group hands a message to whoever asks first, with no affinity at all. So a consumer group does not keep multiple messages from the same user in order on the same worker — say this yourself and you steer into ground you have prepared.
    5. Expect: how do you name consumers? Random names orphan the unacked messages of the previous name after a restart, recoverable only via XAUTOCLAIM. Either use stable ordinals from a stateful deployment, or rely on XAUTOCLAIM and periodically prune dead names with XGROUP DELCONSUMER.
    6. Expect: how do you preserve per-user order? Shard above the bus — hash the user id onto a fixed number of shards and let one consumer own a shard at a time. The consumer group cannot do this for you.

    分析过程 · 先想清楚再作答

    1. 这题是概念题，区分度在于你有没有把「组」和「消费者」两层分清。只答「多个消费者一起消费」会被追着问「那同一条消息会不会被消费两次」，而这正是两层的区别所在。
    2. 先给两层结构：流本身只增不减，组挂在流上、维护一个读游标和一份 pending 清单，消费者挂在组上、只是组内的一个名字。同一个组内的消费者分摊消息（一条只进一个人），不同的组各自都能读到全量——工作队列和发布订阅就是这一个数据结构的两种用法。
    3. 接着点出 pending 清单（PEL）记了哪三件事：这条消息归哪个消费者、被投递过几次、最后一次投递在什么时刻。这三列分别对应「谁在处理」「要不要判成毒消息」「能不能被别人接手」，答出来就说明你真的读过文档而不只是抄过示例。
    4. 结论要落到分配规则上：组把消息分给谁，完全取决于谁先来问，没有任何亲和性。所以 consumer group 天然不保证「同一个用户的多条消息按顺序被同一个人处理」——这一句是把话题引向自己准备好的深水区。
    5. 可以预期的追问一：消费者的名字该怎么取？答：随机名会让进程重启后老名字下的未确认消息变成孤儿，只能靠 XAUTOCLAIM 捡回来，所以要么用有状态部署给的稳定序号，要么就必须依赖 XAUTOCLAIM 兜底，并定期用 XGROUP DELCONSUMER 清理不会再回来的名字。
    6. 可以预期的追问二：怎么保住同一个用户的顺序？答：在总线之上做分片——把用户 id 哈希到固定数量的分片，每个分片同一时刻只由一个消费者持有，顺序就回来了。消费组本身解决不了这件事。

    Key points

    • The stream is append-only; a group holds a read cursor and a pending list; a consumer is a name within a group
    • Within a group messages are split (one message, one consumer); separate groups each get everything, so one structure covers both work queue and pub/sub
    • The pending list records owner, delivery count and last-delivery time — used for takeover, poison detection and timeouts
    • Dispatch has no affinity, so per-user ordering is not guaranteed and needs sharding above the bus
    • Random consumer names orphan unacked messages after a restart; use stable names or rely on XAUTOCLAIM plus XGROUP DELCONSUMER cleanup

    答题要点

    • 流只增不减；组挂在流上，维护读游标和 pending 清单；消费者是组内的一个名字
    • 同组内消息被分摊（一条只进一个消费者），不同组各自拿到全量，所以同一个结构同时支持工作队列和发布订阅
    • pending 清单记三件事：归属的消费者、投递次数、最后一次投递时刻，分别用于接手、毒消息判定和超时检测
    • 分配没有亲和性，谁先来问给谁，所以不保证同一个用户的多条消息顺序，要在总线之上做分片
    • 消费者名字随机会在重启后留下孤儿消息，要么名字稳定，要么依赖 XAUTOCLAIM 并清理死名字
  • What problems do XACK and XAUTOCLAIM each solve, and what changes if you XACK before instead of after doing the work?XACK 和 XAUTOCLAIM 分别解决什么问题？XACK 放在业务处理之前和之后有什么区别？
    Common in ChinaCommon overseasIntermediate#message-bus#redis-streams#error-handling

    How to reason about it · think before answering

    1. The hinge is the second half. The first half is documentation; the second asks whether you know that ack timing decides the delivery semantics of the whole system.
    2. Split the two commands: XACK clears a message from the pending list, meaning the work is genuinely finished; XAUTOCLAIM reassigns a pending message that has been idle past a threshold, meaning its previous owner may be dead. One is the normal path, the other is the failure path.
    3. Then answer the timing question categorically: ack-then-work is at-most-once, work-then-ack is at-least-once. In the first, a crash makes the message vanish — it is not in the pending list, so XAUTOCLAIM cannot recover it. In the second, the worst case is duplicate execution, and duplicates can be blocked by idempotency while lost work cannot. Always work first, except for fire-and-forget telemetry.
    4. Add the point most people miss: on failure the correct action is to do nothing and leave the message pending for XAUTOCLAIM. Acking inside the catch block silently discards failures, which is worse than no retry because you no longer know what you lost.
    5. Add the parameter trade-off: the idle threshold must exceed the worst-case normal processing time. Too small and a healthy in-flight message gets stolen and executed twice; too large and recovery is slow. Be explicit that tuning it only lowers the probability of duplicates — the real backstop is a uniqueness constraint on the consumer side.
    6. Expect: why XAUTOCLAIM rather than XCLAIM? XCLAIM needs an XPENDING scan first and then a named claim, with a race in between; XAUTOCLAIM scans and returns a cursor in one command, and is the recommended approach since Redis 6.2.

    分析过程 · 先想清楚再作答

    1. 题眼在后半句。前半句背文档就能答，后半句在考你知不知道 ack 的时机直接决定了整个系统的投递语义——答不出这一点，面试官会判定你没在生产里管过队列。
    2. 先把两个命令的分工说清：XACK 是「销号」，把消息从 pending 清单里删掉，代表这件事真的做完了；XAUTOCLAIM 是「接手」，把闲置超过阈值的 pending 消息改判给另一个消费者，代表原来那个人可能已经死了。一个负责正常收尾，一个负责异常兜底。
    3. 然后回答时机问题，用一句话定性：先 ack 再干活是 at-most-once，先干活再 ack 是 at-least-once。前者进程一崩消息就人间蒸发，pending 清单里查不到、XAUTOCLAIM 也捡不回来；后者最坏是重复执行，而重复可以用幂等挡掉，丢单挡不掉。所以除了埋点日志这类丢一条无所谓的场景，一律先干活再 ack。
    4. 补一个大多数人漏掉的点：处理失败时正确的动作是**什么都不做**，让消息留在 pending 里等 XAUTOCLAIM。很多人会在 catch 里顺手 ack 掉，那等于把失败的消息静默丢弃，比不重试更糟——因为你连丢了什么都不知道。
    5. 再补一条 XAUTOCLAIM 的参数取舍：空闲阈值要大于「一次正常处理的耗时上限」。给太小会把还在正常处理的消息抢走，同一件事被跑两遍；给太大则故障恢复变慢。但要说清，调大阈值只降低重复概率，不消灭重复，兜底始终是消费端的唯一约束。
    6. 可以预期的追问：为什么用 XAUTOCLAIM 而不是 XCLAIM？答：XCLAIM 要你先 XPENDING 查出候选 id 再点名认领，两步之间还有竞态；XAUTOCLAIM 自己扫 pending 并返回游标，一条命令搞定，是 Redis 6.2 之后的推荐做法。

    Key points

    • XACK is the happy-path close-out: it clears the message from the pending list; repeat acks return 0, so it is naturally idempotent
    • XAUTOCLAIM is the failure backstop: it reassigns pending messages idle past a threshold, answering 'what happens to work held by a dead consumer'
    • Ack-before-work is at-most-once and loses work on a crash; work-before-ack is at-least-once and at worst duplicates, which idempotency can absorb
    • Never ack on failure — leave the message pending for takeover; acking in the catch block silently discards failures
    • The idle threshold should exceed worst-case processing time, but tuning it only reduces duplicates; uniqueness constraints are the real guarantee

    答题要点

    • XACK 负责正常收尾：把消息从 pending 清单里销号，代表这件事真的做完了；重复 ack 返回 0，天生幂等
    • XAUTOCLAIM 负责异常兜底：把闲置超过阈值的 pending 消息改判给另一个消费者，解决「消费者死了它手上的消息怎么办」
    • 先 ack 再干活是 at-most-once，崩溃就丢单；先干活再 ack 是 at-least-once，最坏是重复，可以用幂等挡
    • 处理失败时不要 ack，让消息留在 pending 里等接手；在 catch 里顺手 ack 等于静默丢弃失败
    • 空闲阈值要大于正常处理耗时的上限，但调大只降低重复概率，兜底仍是消费端唯一约束
  • What is at-least-once delivery, and given that messages get redelivered, how do you actually make the business side idempotent?什么是 at-least-once？既然消息会被重复投递，业务上到底要怎么保证幂等？
    Common in ChinaCommon overseasDeep dive#message-bus#idempotency#reliability

    How to reason about it · think before answering

    1. This is the question that gets probed hardest. Most candidates say 'at-least-once, so make the business idempotent' and stop — but the follow-up is exactly what matters: which line of code enforces it.
    2. Explain why the duplicate cannot be removed: committing the business write and acking are two writes to two systems (say Postgres and Redis), so there is always a crash window between finishing the work and XACK. The window can shrink but not disappear, which is why exactly-once is not something the bus gives you.
    3. That yields a sentence worth saying out loud: exactly-once is an effect produced by consumer-side idempotency, not a capability provided by the broker. Kafka transactions achieve it inside a read-Kafka-write-Kafka loop, but the moment the sink is a database or third-party API you are back to at-least-once.
    4. Now name the concrete guards and what each one blocks. First, a unique constraint on runs.idempotency_key with insert ... on conflict do nothing, which blocks duplicate submissions: on conflict the gateway returns the existing run id and never publishes a second bus message. Second, unique(run_id, seq) on the messages table, also on conflict do nothing, which blocks duplicate execution: even if two consumers finish the same run simultaneously the user sees one reply. A cheap short-circuit can sit in between — read the run first and just re-ack if it is already done — but that saves money; correctness comes from the two constraints.
    5. Then the step people get wrong: deriving the key. It must be reproducible from the same intent. Generating a fresh uuid on every retry is the classic mistake, because every retry becomes a new intent and the constraint never fires. The client should mint the key once and reuse it across retries; a server-side fallback can hash session id plus message body plus a second-resolution timestamp.
    6. Expect: what about irreversible side effects such as issuing a refund? Push the idempotency key into the external call (most payment gateways accept an idempotency key header), and record an 'initiated' row locally before calling so the same key deduplicates. For APIs with no such support, fall back to a local state machine plus reconciliation, and say plainly that you would move such operations off the automatic retry path.

    分析过程 · 先想清楚再作答

    1. 这题是本章最容易被追到底的一道。绝大多数人能说出「至少一次，所以业务要幂等」，然后就没有下文了——面试官等的恰恰是下文：幂等具体落在哪一行代码上。答不出具体落点，前半句就是背的。
    2. 先解释为什么消费不掉这个重复：写业务和销号是两个系统的两次写（比如 Postgres 加 Redis），处理完成到 XACK 之间必然存在一个可以崩溃的窗口，崩在那里消息就会被重投。这个窗口只能变小，不能消失，所以 exactly-once 不是总线给你的语义。
    3. 由此得到一句可以直接说出口的结论：exactly-once 是消费端幂等做出来的**效果**，不是中间件提供的**能力**。Kafka 的事务能在「读 Kafka 写 Kafka」的闭环里做到，一旦下游是数据库或第三方 API 就又退回至少一次。
    4. 然后给具体落点，两道闸门要分清各自挡什么：第一道是 runs 表 idempotency_key 上的唯一约束，配 insert on conflict do nothing，挡的是**客户端重复提交**——冲突时接入层直接返回已有的 runId，连总线都不投第二遍；第二道是 messages 表的 unique(run_id, seq)，同样 on conflict do nothing，挡的是**同一条总线消息被执行两遍**，就算两个消费者真的同时跑完，用户也只会看到一条回复。中间还可以加一道便宜的短路：捞到消息先看 run 是不是已经 done，是就直接补一个 XACK 走人——但那是省钱的优化，正确性靠的是那两个唯一约束。
    5. 接着讲最容易做错的一步：幂等键怎么取。它必须能从「同一个意图」稳定推出来。客户端每次重试都新生成一个 uuid 是最常见的错法，那每次都是新意图，唯一约束一次都命中不了，闸门形同虚设。正确做法是客户端生成一次、重试复用同一个值，服务端兜底可以用「会话 id 加消息内容哈希加秒级时间戳」。
    6. 可以预期的追问：不可逆的副作用怎么办，比如发一次退款？答：把外部调用也变成带幂等键的（大多数支付网关都支持 idempotency key 头），并且先在本地库里落一条「已发起」记录再调用，用同一个键去重；实在不支持的接口就只能靠本地状态机加人工对账，这时要主动说出「这类操作我会把它挪出重试路径」。

    Key points

    • At-least-once means a message is processed one or more times, because the business commit and the XACK are two writes to two systems with an unavoidable crash window
    • Exactly-once is an effect of consumer-side idempotency, not a broker feature; any database or third-party sink puts you back at at-least-once
    • Guard one: a unique constraint on runs.idempotency_key with on conflict do nothing blocks duplicate submissions and skips publishing a second bus message
    • Guard two: unique(run_id, seq) on messages with on conflict do nothing blocks duplicate execution, so the user sees exactly one reply
    • The idempotency key must be derivable from the same intent and reused across retries; minting a new uuid per retry defeats the whole mechanism
    • For irreversible side effects, pass the idempotency key through to the external API and record an initiated row locally before calling

    答题要点

    • at-least-once：消息至少被处理一次、可能多次，因为业务提交和 XACK 是两个系统的两次写，中间的崩溃窗口消不掉
    • exactly-once 是消费端幂等做出来的效果，不是中间件的能力；下游只要是数据库或第三方 API 就退回至少一次
    • 闸门一：runs.idempotency_key 唯一约束 + on conflict do nothing，挡客户端重复提交，冲突时不再投递总线消息
    • 闸门二：messages 表 unique(run_id, seq) + on conflict do nothing，挡同一条消息被执行两遍，用户只会看到一条回复
    • 幂等键必须从同一个意图稳定推导，客户端重试要复用同一个值；每次重试新生成 uuid 等于没有幂等
    • 不可逆副作用要把幂等键透传给外部接口，并先落一条本地记录再调用
  • How do you choose between Redis Streams and Kafka, and when is Streams clearly not enough?Redis Streams 和 Kafka 该怎么选？什么情况下 Streams 明显不够用？
    Common in ChinaCommon overseasIntermediate#message-bus#redis-streams#architecture

    How to reason about it · think before answering

    1. The bad answer is 'it depends on volume'. Throughput is never the first criterion — a single Redis node handles tens of thousands of XADDs per second, and most workloads never approach that ceiling. 'Small volume Streams, large volume Kafka' reads as never having run a real evaluation.
    2. Use two real criteria instead: how long the messages must be retained, and whether a second class of consumer will appear. If a message is useless once executed and the execution layer is the only consumer, Streams is plenty and saves an entire operational surface. If you need replay from any point in the last three months, or the same data must feed real-time execution, an offline warehouse and a risk engine, choose Kafka.
    3. Add three structural differences: Streams is memory-first with retention you enforce yourself via MAXLEN or XTRIM, while Kafka does sequential disk writes and keeps weeks by default; a Streams group takes any number of consumers, while Kafka consumers are capped by partition count and extras idle; ordering granularity differs — Streams orders a single stream but dispatches randomly within a group, Kafka pins a key to a partition and orders within it.
    4. Then volunteer the line that shows real depth: Redis persistence is lossy. AOF fsyncs once per second by default, so the last second of writes can vanish, and replication is asynchronous, so a failover can drop unreplicated messages. Using Streams therefore requires a source of truth elsewhere — here the Postgres runs table, with the stream acting only as a trigger; a lost message leaves the run pending and a sweeper republishes it. Treating the bus as the only datastore is the dangerous misuse.
    5. Land on a reusable rule: Streams suits triggering work, Kafka suits data pipelines. One carries one-shot commands, the other carries facts that many parties re-read.
    6. Expect: what about RabbitMQ or SQS? RabbitMQ wins on complex routing and delayed delivery (Streams has no native delay, you republish with a next-eligible timestamp); SQS wins on zero operations at the cost of replay and strict ordering (FIFO queues aside). Framing the criteria as retention, number of consumers, routing complexity and operational budget beats reciting product specs.

    分析过程 · 先想清楚再作答

    1. 这题的坏答案是「看数据量」。吞吐从来不是第一判据——单机 Redis 每秒几万条 XADD 毫无压力，绝大多数业务的量级根本碰不到天花板。答成「量小用 Streams、量大用 Kafka」会被认为没做过选型。
    2. 换成两个真正的判据来推：一、这些消息需要保留多久；二、会不会有第二类消费方。生命周期是「执行一次就没用了」、且只有执行层这一个消费方，Streams 完全够用，还省掉一整套运维；需要「三个月内任意时间点重放」、或者同一份数据要同时喂给实时执行、离线数仓、风控三条链路，那就该上 Kafka。
    3. 再补三条结构性差异：Streams 是内存为主、保留全靠你自己 MAXLEN 或 XTRIM，Kafka 是磁盘顺序写、保留几周是常态；Streams 一个组里加多少消费者都行，Kafka 的消费者数受分区数限制，多了就有人空转；顺序保证的粒度不同，Streams 是单条流内有序而组内分配随机，Kafka 是同 key 落同分区、分区内有序。
    4. 然后主动说出那条最能体现深度的话：Redis 的持久化是有损的。AOF 默认每秒刷盘，最坏丢最后一秒的写入；主从异步复制，故障切换时未同步的消息会消失。所以用 Streams 时架构上必须有一个真相之源——本课是 Postgres 的 runs 表，流只是触发器，丢了消息那个 run 还停在 pending，补投任务会把它捡回来。把总线当唯一数据源是最危险的误用。
    5. 结论落成一句可复用的判断：Streams 适合「触发执行」，Kafka 适合「数据管道」。前者的消息是一次性的命令，后者的消息是需要被多方反复读取的事实。
    6. 可以预期的追问：那 RabbitMQ、SQS 呢？答：RabbitMQ 强在复杂路由和延迟队列（Streams 没有原生延迟投递，要自己带「下次可执行时间」重投）；SQS 强在零运维，代价是没有回放、也没有严格顺序（FIFO 队列另算）。把判据说成「保留时长、消费方数量、路由复杂度、运维预算」四条，比背产品参数强得多。

    Key points

    • The first criterion is not throughput but retention length and whether a second class of consumer will exist
    • One consumer class and messages that expire on execution: Streams is enough, and you probably already run Redis
    • Long retention with arbitrary replay, or one dataset feeding several downstream pipelines: pick Kafka
    • Structural differences: Streams is memory-first with self-managed trimming and random in-group dispatch; Kafka is sequential-disk, key-partitioned with in-partition ordering, and caps consumers at partition count
    • Redis persistence is lossy (per-second AOF fsync, async replication), so the database must be the source of truth with the stream as a trigger plus a republish sweeper
    • One-line rule: Streams triggers work, Kafka moves data

    答题要点

    • 第一判据不是吞吐，是「消息要保留多久」和「会不会有第二类消费方」
    • 只有执行层一个消费方、消息执行完即失效：Streams 够用，且大概率你已经有 Redis，零新增运维
    • 需要长期保留与任意时间点回放、或多条下游链路共用同一份数据：选 Kafka
    • 结构差异：Streams 内存为主、保留靠自己裁剪、组内分配随机；Kafka 磁盘顺序写、按 key 分区且分区内有序、消费者数受分区限制
    • Redis 持久化有损（AOF 每秒刷盘、异步复制），所以真相之源必须是数据库，流只当触发器，靠补投任务兜底
    • 一句话判断：Streams 适合触发执行，Kafka 适合数据管道
  • What do you do with a message that keeps failing? Design a poison-message isolation mechanism.一条消息反复处理失败怎么办？请设计一个毒消息隔离机制。
    Common in ChinaCommon overseasIntermediate#message-bus#error-handling#reliability

    How to reason about it · think before answering

    1. This question probes whether you have ever watched one bad message stall an entire stream. The test is simple: does your answer contain a concrete threshold and a concrete place where isolation happens? If not, you are talking theory.
    2. Describe the failure mode first: under at-least-once you do not ack on failure, so the message stays pending and gets redelivered. A message that fails for everyone therefore loops forever — delivered, failed, idle timeout, claimed, failed — never recovering while continuously consuming worker capacity.
    3. Then give the mechanism, three actions and all of them required. One, use the delivery count the pending list already tracks rather than building a counter table. Two, past the threshold (three deliveries in this course) move the message to a dead-letter stream carrying the original id, delivery count and failure reason. Three, XACK the original stream and mark the run failed with the error recorded. Moving without acking leaves it pending for another takeover; acking without moving makes both the message and its reason disappear, leaving the user stuck on 'thinking'.
    4. Justify the threshold: one delivery kills messages that a single network blip would have let through; ten wastes ten executions of money and time on a message that can never succeed. Three deliveries, spaced by the idle threshold, survives almost all transient faults.
    5. Volunteer a limitation: Redis Streams has no native exponential backoff — redelivery timing is governed by the idle threshold. Backoff requires republishing the message with a next-eligible timestamp, which means building a delay queue yourself. Naming this shows you know where Streams ends.
    6. Expect: is creating the dead-letter stream the end of it? No. Its depth must be alerted on, since going from zero to non-zero usually means a class of input your code cannot handle — a real bug, not bad luck. Keep a replay path too: republish the stored fields back to the original stream, and because the idempotency key is preserved, replay cannot cause duplicate execution. Teams that build a dead-letter stream and never open it have simply muted their failures.

    分析过程 · 先想清楚再作答

    1. 这题在考你有没有踩过「一条坏消息拖垮整条流」。判断标准很简单：你的回答里有没有出现一个具体的阈值和一个具体的落地位置，没有就是在讲概念。
    2. 先把故障模式说清楚：按 at-least-once 的规矩，失败就不 ack、留在 pending 等重投，于是一条无论谁来都会失败的消息进入死循环——投递、失败、闲置超时、被接手、再失败。它自己永远好不了，还持续占用消费者的处理能力。
    3. 然后给机制，三个动作缺一不可：一、判定依据用 pending 清单自己记的投递次数，不要另建计数表；二、超过阈值（本课固定 3 次）就把消息搬到一条死信流，字段里带上原始消息 id、投递次数和失败原因；三、对原流 XACK，同时把这次执行标成失败并写入错误原因。只搬不 ack，它还躺在 pending 里等着被接手；只 ack 不搬，消息和失败原因一起消失，用户永远停在「正在思考」。
    4. 阈值的取值要给出权衡：定 1 会让一次网络抖动就把本来能成功的消息判死；定 10 会在一条必死的消息上浪费十次执行的钱和时间。3 次配合每次之间的空闲阈值，足够熬过绝大多数瞬时故障。
    5. 还要主动说出一个缺口：Redis Streams 没有原生的指数退避，重投时机由空闲阈值决定。想要退避就得自己把消息重新投递并带上「下次可执行时间」，那已经是在实现延迟队列了——这一条能体现你知道 Streams 的边界在哪。
    6. 可以预期的追问：死信流建完就完了吗？答：不。死信条数必须接进告警，它从 0 变成非 0 通常意味着有一类输入你的代码处理不了，是真 bug 而不是运气差；还要留一个重放入口——把死信里的字段原样投回原流即可，因为幂等键还在，重放不会产生重复执行。见过团队把死信建起来半年没打开过，那等于把故障静音了。

    Key points

    • Failure mode: under at-least-once you do not ack on failure, so an always-failing message is redelivered forever and keeps consuming worker capacity
    • Use the delivery count already tracked in the pending list rather than a separate counter table
    • Fix the threshold at three deliveries: one kills transient failures, ten wastes ten executions on a message that can never succeed
    • Isolation needs all three actions: move to a dead-letter stream with original id, delivery count and reason; XACK the original stream; mark the run failed with the error stored
    • Redis Streams has no native exponential backoff — redelivery timing follows the idle threshold, so backoff means implementing delayed republishing yourself
    • Alert on dead-letter depth and keep a replay path; the idempotency key survives, so replay cannot duplicate execution

    答题要点

    • 故障模式：at-least-once 下失败不 ack，一条永远失败的消息会无限重投并持续占用消费者
    • 判定依据用 pending 清单里记的投递次数，不需要另建计数表
    • 阈值固定 3 次：定 1 会误杀瞬时故障，定 10 会在必死消息上浪费十次执行成本
    • 隔离动作三件缺一不可：搬到死信流（带原始 id、投递次数、失败原因）、对原流 XACK、把这次执行标成失败并写入原因
    • Redis Streams 没有原生指数退避，重投时机由空闲阈值决定，要退避得自己实现延迟投递
    • 死信流要接告警并留重放入口；幂等键还在，重放不会导致重复执行

Comments