Dayward AIWeChat Mini Program
Build an AI Short-Drama Production Pipeline With Agents in 14 Days
Week 2 · D8🔒 LockedAbout 6 hours

A Workflow Engine: Turning the Pipeline Into a Resumable Task Graph

Replace the sequential script with a workflow engine that has nodes, dependencies, and state, so every step's output lands on disk, every step is idempotent, and a failure resumes from the checkpoint instead of burning money from scratch.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Implement a minimal task-graph executor that schedules nodes by dependency and records each node's state
  2. Make every node idempotent: the same input doesn't spend money twice, and a rerun only completes the unfinished parts
  3. Log each run's outputs and cost into a ledger, traceable by run id

Yesterday's pain report had a number on it: when the fourth stage blew up, the CNY 3.1250 already spent would be spent again in full. Today we turn it into 0. When you finish, scroll back up and tick the three goals off.

Plain-Language Walkthrough

The slate: which take we are on, where a reshoot resumes

Every set has a small board, filled in before each take: scene, shot, take number. It looks like something for post to match material with, but its real job is getting "where are we" out of any one person's head. The DP is replaced mid-shoot, a machine crashes and reboots, pickups happen the next day — check the log and you know where to resume.

Yesterday's pipeline had no slate. It kept progress in the process's call stack — step four meant step four, and when the process died, what the first three steps did, what they cost and where their artifacts were all died with it. So it could only start over.

Fixing that requires something more basic first: separating scheduling, state and retry from the business code.

Yesterday's six stages were six await calls in sequence, with business logic welded to execution order. Now write it differently: each stage declares only what it is, not when it runs.

node-spec.js
// A node declares four things: who I depend on, what my inputs are, which files I produce,
// and how I run. "When do I run", "what if I crash", "have I run before" are not its business.
export const clipsNode = {
  id: 'clips',
  title: 'shot videos',
  deps: ['frames'],
  version: 1, // bump this whenever the node's implementation changes; old artifacts die at once
  inputs: { model: 'video-model-id', shots: shots.map((s) => `${s.id}|${s.visual}|${s.durationSec}`) },
  outputs: shots.map((s) => `${s.id}.mp4`),
  async run(ctx) {
    for (const s of shots) {
      // Read a dependency's artifacts from its directory, never pass them in memory -
      // that is what makes it possible for a node to run in another process.
      const firstFrame = `${ctx.depDir('frames')}/${s.id}.png`
      const r = await video.generate({ prompt: s.visual, firstFrame, outPath: `${ctx.outDir}/${s.id}.mp4` })
      ctx.record({ nodeId: 'clips', kind: 'video', units: s.durationSec, costCny: r.costCny })
    }
    return { note: `${shots.length} shots` }
  },
}

The value of this step is not prettier code; it is that the node can now be pushed around freely by the engine: skipped, reordered, run in another process, or asked before running whether this work has already been done. And that last question is the entire source of the savings.

So how does the engine know the work has already been done? The answer is a key, and whether that key is designed correctly decides whether your pipeline halves its bill or quietly hands you an episode with the wrong footage in it.

Sign in and link WeChat to read

Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.

Mini program QR code

Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.

Interview questions

  • How do you make a node that calls a paid generation API idempotent? What belongs in the cache key and what does not?怎么让一个会调用付费接口的生成节点是幂等的?缓存键里该放什么、不该放什么?
    Common in ChinaCommon overseasIntermediate#idempotency#caching#workflow-engine

    🔒 Unlock this chapter to see the answer keys.

  • What state must you persist to support resuming a workflow? Is per-node completion status enough?要支持断点续跑,你需要持久化哪些状态?只存每个节点的完成状态够不够?
    Common in ChinaCommon overseasDeep dive#workflow-engine#state-persistence#resume

    🔒 Unlock this chapter to see the answer keys.

  • When should you write your own scheduler, and when should you adopt an off-the-shelf workflow engine?什么时候该自己写调度,什么时候该直接上现成的工作流引擎?
    Common in ChinaCommon overseasBasic#architecture#build-vs-buy#workflow-engine

    🔒 Unlock this chapter to see the answer keys.

Comments