Dayward AIWeChat Mini Program
From Frontend Engineer to Agent Engineer in 30 Days
Week 2 · D14🔒 LockedAbout 6 hours

Deployment and Operations: Multi-Worker Compose, Heartbeats, Health Checks, Graceful Shutdown, Dev/Prod Isolation; Week Two Retrospective

Run mini-koda as a multi-worker deployment with docker compose, add heartbeats, health checks, and graceful shutdown, sort out how dev and prod stay isolated, and wrap up week two.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Scale the worker to multiple instances with docker compose and observe how traffic is split
  2. Implement worker heartbeat reporting and a gateway health-check endpoint
  3. Implement graceful shutdown: finish the task in hand before exiting once a stop signal arrives

Yesterday closed on this: the scheduler, the bus, the worker, the database, and the ledger all run in one terminal on your machine, and you find problems by reading logs. Today it becomes a deployment somebody else could take over, and then we string week two into one line.

Plain-Language Walkthrough

One image, three shop assistants: multi-replica workers in compose

The convenience store downstairs is open around the clock, and not because of any one assistant — one set of procedures, three shifts rotating, and whoever is on can take payments, restock, and count inventory. A shop is a role plus a rota, not a person. Define the role clearly and people become replaceable.

D8 through D13 already defined the roles: the gateway is stateless, the worker takes work off the bus, and all state lives in Postgres and Redis. Scaling replicas was architecturally finished long ago; today only declares it.

The declaration lives in a compose file. Its division of labor differs from D7's Dockerfile: a Dockerfile answers how this one process starts, and compose answers how these processes assemble into a system. D7's image does not change one line today.

YAMLYAML
services:
  gateway:
    build: .
    environment:
      ROLE: gateway            # one image, two roles, branching on an environment variable
      APP_ENV: dev
      REDIS_URL: redis://redis:6379
    ports: ['3014:3014']       # only this one needs a port exposed
    depends_on:
      postgres: { condition: service_healthy }
      redis: { condition: service_healthy }
    stop_grace_period: 25s
 
  worker:
    build: .                   # the same image
    environment:
      ROLE: worker
      APP_ENV: dev
      REDIS_URL: redis://redis:6379
    deploy:
      replicas: 3              # three replicas, each its own container and hostname
    stop_grace_period: 25s     # must exceed the 20-second grace in the code, see section four

Four decisions deserve explanation.

One image, not two. The gateway and the worker already share a repository, a schema, and configuration. Two images means two pipelines and two version numbers, and produces the hardest failure to diagnose: the gateway on the new version with the worker still on the old. Branch on one environment variable and versions align by construction.

Worker replicas need no port mapping. They accept no inbound connections and go to the bus for work. So adding a replica touches no load balancer, no service registry, and no routing — a new container comes up, issues an XREADGROUP, and has work; whereas every extra gateway must be attached behind a load balancer. Pull scales more easily than push, a downstream benefit of D9's choice of a message bus.

Replica names need thought. Three replicas share one set of environment variables, so the consumer name cannot live there. This course uses the container hostname, which is naturally one per container. The cost is the trap D9 named: rebuilding a container changes the name, unacknowledged messages under the old one become orphans, and only XAUTOCLAIM recovers them.

The replica count has a ceiling, and it is not CPU. Connections equal replicas times pool size, and Postgres's default max_connections is only 100 — 10 replicas with 10 connections each saturate it, and the eleventh gets "too many clients." Count connections before scaling; it is the first wall you hit going from three replicas to thirty, and the error has nothing to do with your business code.

At this point docker compose up -d brings up a multi-replica deployment and docker compose ps faithfully shows three Up. But Up says only that the process exists — it may be stuck in a garbage collection doing nothing. And the next release replaces all three containers, so how do you guarantee no in-flight conversation is cut off?

Sign in and link WeChat to read

Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.

Mini program QR code

Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.

Interview questions

  • With multiple replicas, how do you design heartbeats and health checks? Are they the same thing?多实例部署下,怎么设计心跳和健康检查?两者是同一件事吗?
    Common in ChinaCommon overseasIntermediate#observability#deployment#distributed-systems

    🔒 Unlock this chapter to see the answer keys.

  • What is graceful shutdown, and why is killing a process outright risky? Walk through the steps.什么是优雅停机?为什么直接 kill 进程有风险?请说出具体步骤。
    Common in ChinaCommon overseasIntermediate#deployment#reliability#operations

    🔒 Unlock this chapter to see the answer keys.

  • During a rolling deploy, how do you keep in-flight tasks from being interrupted?滚动发布时,如何避免正在处理的任务被打断?
    Common in ChinaCommon overseasIntermediate#deployment#reliability#operations

    🔒 Unlock this chapter to see the answer keys.

  • How do you isolate dev from prod so local development cannot touch production data?怎么设计 dev 与 prod 的隔离,防止本地开发影响线上数据?
    Common in ChinaCommon overseasBasic#operations#security#configuration

    🔒 Unlock this chapter to see the answer keys.

  • System design: design an IM agent platform where users chat with an AI assistant inside a messaging app. The assistant calls tools, remembers long-term preferences, and proactively pushes scheduled messages. Target 100k daily active users.系统设计:请设计一个 IM Agent 平台——用户在即时通讯软件里和一个 AI 助手对话,助手能调用工具、记住长期偏好、还能定时主动推送。要求支撑十万日活。
    Common in ChinaCommon overseasDeep dive#system-design#distributed-systems#cost#operations

    🔒 Unlock this chapter to see the answer keys.

Comments