One Episode Wrapped: Stringing Six Stages Into an End-to-End Pipeline and Tallying the First Bill
String script, assets, shots, voiceover, and assembly into one line a single command can run end to end, measure the time and cost of each stage, and find the pipeline's three most fragile points right now.
Today's Goals
- Produce one complete episode from a single sentence with one command, and know exactly where it's stuck if any step fails
- Measure the time and cost of each stage, and point out where the bottleneck is
- List the pipeline's three fragile points right now, and say which day of week two fixes each one
Over six days you got six stages running individually, but your hands still sit between them: after the script you copy the JSON into the next script by hand; after dubbing you check by hand that the files exist. Today, take your hands off. When you finish, scroll back up and tick the three goals off.
Plain-Language Walkthrough
The day an episode wraps is when the problems start
Film crews call the last shot the wrap, and everyone applauds. Anyone who has done post knows the problems start at that moment: material scattered across three drives with different naming, the sound recordist's timecode two frames off from camera, an NG take whose filename differs from the good one by a single letter.
String six stages together and you hit exactly those things. None of them existed while you debugged each stage alone, because your brain was doing the handoffs: you knew where the last script wrote its files, you knew which JSON to feed the next step. Connected up, that knowledge has to move out of your head and into the code.
The first thing to bite is artifact paths. Running the image script alone, output/first-frame.png is perfectly fine. Connected up, the second run overwrites the first one's output; a mid-run failure leaves you unable to tell which files on disk belong to this attempt; and comparing "did changing the sentence make it better" is impossible because both runs' artifacts are mixed together.
There is one fix, and the rule has to be laid down from the start: allocate a run identifier per run and hang every artifact under it.
work/
โโโ runs/run-20260907-001/
โโโ run.json this run's metadata and cost ledger
โโโ script/ script, character cards
โโโ assets/ character look tests
โโโ shots/<shotId>/ first frame, shot video, voice
โโโ timeline/ the timeline table
โโโ output/ cut, subtitles, coverThe layout looks plain, but it settles three problems at once: two runs never overwrite each other; leftovers from a failure are obvious; and reproducing a result is a matter of zipping one directory and sending it to a colleague.
// One run's directory layout. The whole pipeline gets paths through this object;
// any place that concatenates a path string directly is a latent bug.
export class Run {
constructor(id) {
this.id = id
this.dir = `work/runs/${id}`
this.costs = []
}
path(...parts) {
return [this.dir, ...parts].join('/')
}
// Record spend the moment it happens, not by tallying after the run finishes -
// a tally done afterwards is always wrong when the run failed.
record(entry) {
this.costs.push({ ...entry, runId: this.id, at: new Date().toISOString() })
}
}
export function newRunId(now = new Date()) {
const p = (n, w = 2) => String(n).padStart(w, '0')
const stamp = `${now.getFullYear()}${p(now.getMonth() + 1)}${p(now.getDate())}`
return `run-${stamp}-${p(Math.floor((now.getHours() * 60 + now.getMinutes()) % 1000), 3)}`
}# One run's directory layout. The whole pipeline gets paths through this object;
# any place that concatenates a path string directly is a latent bug.
from datetime import datetime
class Run:
def __init__(self, run_id: str):
self.id = run_id
self.dir = f"work/runs/{run_id}"
self.costs: list[dict] = []
def path(self, *parts: str) -> str:
return "/".join([self.dir, *parts])
# Record spend the moment it happens, not by tallying after the run finishes -
# a tally done afterwards is always wrong when the run failed.
def record(self, entry: dict) -> None:
self.costs.append({**entry, "runId": self.id, "at": datetime.now().isoformat()})
def new_run_id(now: datetime | None = None) -> str:
now = now or datetime.now()
seq = (now.hour * 60 + now.minute) % 1000
return f"run-{now:%Y%m%d}-{seq:03d}"The second problem is log flooding. Six stages each print their own logs, one command scrolls hundreds of lines past, and when something breaks you cannot tell which stage broke it. Today's logging rule is simple too: each stage prints one line on start and one line on finish with its elapsed time and artifact count, and pushes the rest into files. What stays on the terminal should be a progress table you can scan at a glance, not a running diary.
So what most deserves measuring once things are connected? Not "did it run," but how much time and how much money each step took. Those two numbers decide what you fix first next week โ and they will most likely overturn your intuition.
Interview questions
In a multi-step generation pipeline, one step fails. What behavior do you want the system to have?ไธๆกๅคๆญฅ้ชค็็ๆๆตๆฐด็บฟ๏ผไธญ้ดๆไธๆญฅๅคฑ่ดฅไบ๏ผไฝ ๅธๆ็ณป็ปๆไปไน่กไธบ๏ผ
Common in ChinaCommon overseasIntermediate#pipeline-reliability#idempotency#error-handling๐ Unlock this chapter to see the answer keys.
How do you measure the cost of a generation pipeline, and what besides money should you measure?ๆไนๅบฆ้ไธๆก็ๆๆตๆฐด็บฟ็ๆๆฌ๏ผ้คไบ้ฑ่ฟ่ฆ้ไปไน๏ผ
Common in ChinaCommon overseasBasic#observability#cost-accounting#pipeline-design๐ Unlock this chapter to see the answer keys.
After chaining several individually working steps into one pipeline, which problems appear that single-step debugging never shows?ๆๅคไธชๅทฒ็ปๅ่ช่ท้็็ฏ่ไธฒๆไธๆกๆตๆฐด็บฟไนๅ๏ผๅชไบ้ฎ้ขๆฏๅ็ฌ่ฐ่ฏๆถ็ไธ่ง็๏ผ
Common in ChinaCommon overseasDeep dive#integration#pipeline-design#observability๐ Unlock this chapter to see the answer keys.
