The Run State Machine, Streaming Output Back, Ordering by runId, SSE Waiters, Merging Interruptions Within 30 Seconds
Build a run state machine for every conversation turn, stream the worker's output back in order by runId to the waiting SSE connection, and merge interruptions that arrive within 30 seconds.
Today's Goals
- Draw a run's complete state machine from creation to completion (including failure and interruption states)
- Stream a worker's output back in order by runId to a waiting SSE client
- Merge a user interruption within 30 seconds into the same input, rather than opening a second run
D10 preserved order on the execution side, and from the user's side it is still "sent it and heard nothing." Today we connect the output back, and only then is this chain complete.
Plain-Language Walkthrough
Draw the line between today and D7 first, because the whole chapter stands on it. On D7 one process both generated and pushed: the model emitted a character and the same call stack immediately wrote a frame, with no gap between generating and pushing, so order was simply not a problem โ it was correct by construction. Today generation is in the worker and pushing is in the gateway, with a message bus between them: characters are produced in one process while the connection hangs off another, and the bus promises nothing about when or in what order you receive them. D7 covered the server side of SSE thoroughly (four headers, the : ping heartbeat, disconnect handling) and none of it is repeated; all the effort goes into the four new problems that gap creates: state, order, multiple subscribers, and reconnection.
An execution needs an identity: the run state machine
A radio station makes an episode: commissioned, recorded in studio, broadcast, archived. Ask at any moment which step the episode is at and there is a definite answer, and the steps are one-way โ broadcast cannot revert to being recorded.
In the single-process era you did not need this. Inside D7's POST /chat, "how far the execution has got" was the call stack itself: running meant running, returned meant finished, thrown meant failed. That state lived in process memory, unnamed, and needed no name.
After the split it must have a name, because at least three parties have to answer the same question at once: the gateway needs to know whether this execution is still alive before deciding whether to hold an SSE connection; the worker needs to know whether somebody is already doing this message before deciding to execute; and when the user reopens the page the frontend needs to know whether the previous question is still generating. Those three are not in one process, and a table is the only thing that can align them. That is D8's runs table and its six statuses:
+---------------------------- normal path -------------------------+
| |
+---------+ delivery taken +---------+ first fragment +-----------+ v
| pending | ---------------) | running | -------------) | streaming |-) +------+
+---------+ +---------+ +-----------+ | done |
| | | +------+
| v v
| +--------+ +--------+
+------------------) | failed | | failed | retries exhausted
+--------+ +--------+
| | |
+------------------) +-----------+ (--------------+
| cancelled | merged away or cancelled by the user
+-----------+Four normal statuses run forward and two abnormal exits are available at any time: failed is retries exhausted, cancelled is merged away or cancelled by the user. running and streaming are separate because they mean entirely different things to the user: running is "a worker has taken it, and not one character yet," and streaming is "the first character is out." That boundary is the observation point for time-to-first-token, and the basis on which the frontend decides between keeping the spinner and starting the typewriter.
A state machine's value is not listing those words, it is putting the illegal transitions into code where they are blocked:
// A transition table rather than a chain of ifs: it can be read, tested, and pasted
// straight into a design document
const TRANSITIONS = {
pending: ['running', 'cancelled', 'failed'],
running: ['streaming', 'done', 'failed', 'cancelled'],
streaming: ['done', 'failed', 'cancelled'],
done: [], // a terminal state has no outgoing edges, the most valuable line in the table
failed: [],
cancelled: [],
}
function transition(from, to) {
if (!TRANSITIONS[from].includes(to)) {
throw new Error(`illegal state transition: ${from} -> ${to}`)
}
return to
}from enum import StrEnum
class RunStatus(StrEnum):
PENDING = "pending"
RUNNING = "running"
STREAMING = "streaming"
DONE = "done"
FAILED = "failed"
CANCELLED = "cancelled"
# frozenset says "this table will not be modified", which fits the meaning better than a list
TRANSITIONS: dict[RunStatus, frozenset[RunStatus]] = {
RunStatus.PENDING: frozenset({RunStatus.RUNNING, RunStatus.CANCELLED, RunStatus.FAILED}),
RunStatus.RUNNING: frozenset({RunStatus.STREAMING, RunStatus.DONE, RunStatus.FAILED, RunStatus.CANCELLED}),
RunStatus.STREAMING: frozenset({RunStatus.DONE, RunStatus.FAILED, RunStatus.CANCELLED}),
RunStatus.DONE: frozenset(),
RunStatus.FAILED: frozenset(),
RunStatus.CANCELLED: frozenset(),
}
def transition(current: RunStatus, target: RunStatus) -> RunStatus:
if target not in TRANSITIONS[current]:
raise ValueError(f"illegal state transition: {current} -> {target}")
return target// The idiomatic Java state machine hangs the outgoing edges off the enum itself, stored in
// an EnumSet - a bitmap implementation, constant-time lookups, and forgetting to give a
// new status its edges is caught at compile time
enum RunStatus {
PENDING, RUNNING, STREAMING, DONE, FAILED, CANCELLED;
Set<RunStatus> allowedNext() {
return switch (this) {
case PENDING -> EnumSet.of(RUNNING, CANCELLED, FAILED);
case RUNNING -> EnumSet.of(STREAMING, DONE, FAILED, CANCELLED);
case STREAMING -> EnumSet.of(DONE, FAILED, CANCELLED);
case DONE, FAILED, CANCELLED -> EnumSet.noneOf(RunStatus.class);
};
}
RunStatus to(RunStatus target) {
if (!allowedNext().contains(target)) {
throw new IllegalStateException("illegal state transition: " + this + " -> " + target);
}
return target;
}
}// Swift uses an exhaustive switch: add a status to the enum later and this switch stops
// compiling, so the compiler forces you back to fill in its edges - the strongest
// protection of the four languages against forgetting a status.
// The edge table matches D8's exactly (including pending to failed). Only the method
// differs: D8 gave canTransition(to:) -> Bool, and today it becomes transition(to:) throws -
// a state machine that really blocks illegal transitions cannot return a Bool a caller
// can quietly ignore
enum RunStatus: String, Codable {
case pending, running, streaming, done, failed, cancelled
var allowedNext: Set<RunStatus> {
switch self {
case .pending: [.running, .cancelled, .failed]
case .running: [.streaming, .done, .failed, .cancelled]
case .streaming: [.done, .failed, .cancelled]
case .done, .failed, .cancelled: []
}
}
func transition(to target: RunStatus) throws -> RunStatus {
guard allowedNext.contains(target) else {
throw RunError.illegalTransition(from: self, to: target)
}
return target
}
}What that throw buys is a class of incident blocked at the moment of writing. In a distributed system "a run that is already done receives another fragment" is routine rather than exceptional โ the bus delivers at least once (D9), one network wobble replays an old message, and a lagging worker may wake after wrapping up and write once more. Without the table that write lands quietly, the user sees half a stray sentence appended to their reply, and you search the logs for ages without finding who wrote it. With the table it throws on the spot and lands in the log.
A state machine's real product is not those words, it is the discipline that every status write must go through this function. Bypass it once with an UPDATE runs SET status = 'done' and the table degrades into a comment.
Which raises the second problem: the worker generates characters one at a time inside its own process, so how do those characters get back, in order, to the connection still hanging off the gateway?
Interview questions
How would you design the state machine for one agent run, and which failure states must it cover?ๆไน่ฎพ่ฎกไธๆฌก Agent ๆง่ก๏ผrun๏ผ็็ถๆๆบ๏ผ้่ฆ่ฆ็ๅชไบๅผๅธธ็ถๆ๏ผ
Common in ChinaCommon overseasBasic#state-machine#distributed-systems๐ Unlock this chapter to see the answer keys.
Several clients subscribe to the same run's streaming output at once. How do you guarantee each of them receives the full content in order?ๅคไธชๅฎขๆท็ซฏๅๆถ่ฎข้ ๅไธๆฌกๆง่ก็ๆตๅผ่พๅบ๏ผๆไนไฟ่ฏๆฏไธชๅฎขๆท็ซฏ้ฝๆถๅฐๅฎๆดไธๆๅบ็ๅ ๅฎน๏ผ
Common in ChinaCommon overseasIntermediate#sse#ordering#fan-out๐ Unlock this chapter to see the answer keys.
A user sends another message while the agent is still answering the previous one. How should the system handle it?็จๆทๅจ Agent ่ฟๆฒกๅๅคๅฎ็ๆถๅๅๅๆฅไธๆกๆถๆฏ๏ผๅบ่ฏฅๆไนๅค็๏ผ
Common in ChinaCommon overseasIntermediate#interrupt-merge#state-machine#cost๐ Unlock this chapter to see the answer keys.
After a streaming client reconnects, how do you deliver every missed chunk exactly once โ no gaps, no duplicates?ๆตๅผๆฅๅฃ็ๅฎขๆท็ซฏๆญ็บฟ้่ฟๅ๏ผๆไนๅๅฐๆขไธไธข็ๆฎตไนไธ้ๅค๏ผ
Common in ChinaCommon overseasDeep dive#sse#idempotency#reconnect๐ Unlock this chapter to see the answer keys.
