A Redis Streams Message Bus: XADD/XREADGROUP/XACK/XAUTOCLAIM, Consumer Groups, Poison Messages
Build a message bus with Redis Streams, understand how a consumer group lets multiple workers split consumption, and handle a "poison message" that keeps failing.
Today's Goals
- Write a message with XADD, consume it as a consumer group with XREADGROUP, and XACK it
- Explain what problem each of XACK and XAUTOCLAIM solves
- Design a simple isolation mechanism for a poison message (one that keeps failing)
Yesterday split the service into a gateway, a message bus, and a worker, and that middle bus is still just a box on the diagram: POST /chat writes a run into the runs table and returns, and nobody answers who executes it afterwards. Today we fill in that box — and by the end of this chapter the full D8-to-D9 chain genuinely runs for the first time.
Plain-Language Walkthrough
Once it is persisted, who does the work
Think of the parcel pickup point by the entrance to a residential block. The van arrives and the driver does not carry each parcel door to door waiting for you to answer. They shelve it, record a shelf number, and leave; collection is somebody else's job. Shelving and delivery are completely separated by that shelf: the van can keep unloading, and one parcel nobody ever collects does not block anyone behind it.
At the end of D8 your service was stuck in exactly the state of having no shelf. POST /chat authenticates, rate-limits, writes to the database, returns a runId, and stops. The most immediate idea is: the request is already in hand, so just execute it here and return when done. D8 priced that road — one slow execution stalls the whole process's event loop — and here are three harder reasons.
One, the traffic shapes do not match. Users arrive in bursts, and ten minutes of morning peak may equal an hour of the flat period. The intake layer must be provisioned for peak; the execution layer only needs the average, letting backlog queue on the shelf. Cram them into one process and you are forced to provision the execution layer for peak too, paying for processes that idle most of the day.
Two, the failure boundaries do not match. Model wobbles, tool timeouts, one execution running the full 40 seconds — all routine for the execution layer. Share a process with accepting requests and one execution's failure reaches connections mid-handshake, so the user sees "the site is down" when in truth some model was slow.
Three, the scaling units do not match. The intake layer runs out of connections and memory, the execution layer runs out of concurrency and egress bandwidth, and combined you can only add both at once.
Could you skip Redis and use the runs table as the queue? select ... for update skip locked genuinely works, and a dozen messages a second is entirely fine. Its problem is polling: set the interval to 1 second and the user waits up to a second for nothing; set it to 50 milliseconds and an idle database gains 20 pointless queries a second multiplied by the worker count. A message bus's value is push — XREADGROUP's BLOCK parameter lets a worker hang there and wake the instant a message exists.
State the price too: one more component to operate, one more hop of latency, and messages will definitely be delivered more than once — which is section four's subject.
In code today's change is two lines: after persisting, publish the id.
// Dependencies: ioredis 5.x. createRun is D8's insert, and inserted tells you whether
// a row was really added
const { runId, inserted } = await store.createRun({ sessionId, idempotencyKey, input: text })
// Hit the idempotency_key unique constraint: this is a client retry, somebody is already
// running it, so do not publish again
if (!inserted) return reply.send({ runId, duplicated: true })
// The message body carries only ids, not the text: a bus is routing, not a database
const streamId = await redis.xadd('koda:runs', '*', 'runId', runId, 'sessionId', sessionId)
reply.send({ runId, streamId })# Dependencies: redis 5.x's redis.asyncio. xadd takes a dict directly, which beats
# hand-writing a flat array
run_id, inserted = await store.create_run(session_id, idempotency_key, text)
# Hit the idempotency_key unique constraint: this is a client retry, do not publish again
if not inserted:
return {"runId": run_id, "duplicated": True}
# The message body carries only ids, not the text: a bus is routing, not a database
stream_id = await redis.xadd("koda:runs", {"runId": run_id, "sessionId": session_id})
return {"runId": run_id, "streamId": stream_id}// Dependencies: Lettuce (sync API). executeUpdate returning 0 means the
// on conflict do nothing took effect
var created = store.createRun(sessionId, idempotencyKey, text);
if (!created.inserted()) {
return new SubmitResult(created.runId(), true, null);
}
// Lettuce's xadd takes a Map of String to String, which matches Redis's field model exactly
var fields = Map.of("runId", created.runId(), "sessionId", sessionId);
String streamId = redis.xadd("koda:runs", fields);
return new SubmitResult(created.runId(), false, streamId);// Dependencies: RediStack. It has no dedicated Streams methods, so send the raw command -
// the argument order matches exactly what you type into redis-cli, which actually makes
// it easier to check against the documentation
let created = try await store.createRun(sessionId: sessionId, key: idempotencyKey, input: text)
guard created.inserted else {
return SubmitResult(runId: created.runId, duplicated: true, streamId: nil)
}
// RESPValue is not a string literal type, so write the arguments as [String] in redis-cli
// order first and convert them in one pass
let args = ["koda:runs", "*", "runId", created.runId, "sessionId", sessionId]
.map(RESPValue.init(from:))
let streamId = try await redis.send(command: "XADD", with: args).get().string
return SubmitResult(runId: created.runId, duplicated: false, streamId: streamId)Note the order of those two steps: persist first, publish second. Reversed, a worker may well take the message before that runs row has committed and read a runId that does not exist.
Even in the right order, one hole remains: the write succeeds and XADD fails, so that run sits at pending forever with nobody executing it and nobody raising an error. The cheap fix is a periodic job that scans for runs created more than 30 seconds ago and still pending, and republishes them. As long as the bus and the database are two systems, this hole necessarily exists; you can compensate for it, not eliminate it. Volunteering that in an interview is worth more than any amount of fluency about XADD.
So how is the shelf actually used? Redis Streams has only four commands to remember, and together they happen to constitute a complete delivery semantics.
Interview questions
How does a Redis Streams consumer group work, and why can it serve both as a work queue and as pub/sub?Redis Streams 的 consumer group 是怎么工作的?为什么它既能做工作队列又能做发布订阅?
Common in ChinaCommon overseasBasic#message-bus#redis-streams🔒 Unlock this chapter to see the answer keys.
What problems do XACK and XAUTOCLAIM each solve, and what changes if you XACK before instead of after doing the work?XACK 和 XAUTOCLAIM 分别解决什么问题?XACK 放在业务处理之前和之后有什么区别?
Common in ChinaCommon overseasIntermediate#message-bus#redis-streams#error-handling🔒 Unlock this chapter to see the answer keys.
What is at-least-once delivery, and given that messages get redelivered, how do you actually make the business side idempotent?什么是 at-least-once?既然消息会被重复投递,业务上到底要怎么保证幂等?
Common in ChinaCommon overseasDeep dive#message-bus#idempotency#reliability🔒 Unlock this chapter to see the answer keys.
How do you choose between Redis Streams and Kafka, and when is Streams clearly not enough?Redis Streams 和 Kafka 该怎么选?什么情况下 Streams 明显不够用?
Common in ChinaCommon overseasIntermediate#message-bus#redis-streams#architecture🔒 Unlock this chapter to see the answer keys.
What do you do with a message that keeps failing? Design a poison-message isolation mechanism.一条消息反复处理失败怎么办?请设计一个毒消息隔离机制。
Common in ChinaCommon overseasIntermediate#message-bus#error-handling#reliability🔒 Unlock this chapter to see the answer keys.
