Dayward AIWeChat Mini Program
Build an AI Short-Drama Production Pipeline With Agents in 14 Days
Week 1 · D6🔒 LockedAbout 5 hours

The Editing Bay: Assembling Footage Into One Vertical Cut With ffmpeg

Assemble shot footage, voiceover, subtitles, and background audio along a timeline into one vertical finished cut, add transitions and a cover, and have the program generate the command from a structured timeline rather than letting the model improvise it.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Use ffmpeg to splice multiple shots along a timeline into one vertical finished cut
  2. Add subtitles, an audio track, transitions, and a cover to the finished cut, and output encoding parameters the target platform accepts
  3. Have the program generate the ffmpeg command from a structured timeline, rather than handing the command to the model to improvise

You already have a picture for every shot, a voice for every line, and a timeline table. Today does exactly one thing: turn those scattered files into an mp4 you could publish. When you finish, scroll back up and tick the three goals off.

Plain-Language Walkthrough

The editing bay has only one table

An editor's screen is split in half: the preview window on top, a horizontal track laid out below. Clips sit in cells along that track, each labeled from second X to second Y. The editor's job, stripped down, is arranging those cells in order so picture and sound line up at the same instant.

We do exactly that, except our track is not dragged into place, it is computed. D5 already defined it: an array where each record says which shot, from which millisecond to which, which video file, which audio, which subtitle line. That is the timeline, and everything today starts from it.

It looks like this — one episode, four shots, twenty-one seconds:

JSONJSON
[
  {
    "shotId": "s01",
    "startMs": 0,
    "endMs": 6000,
    "clipPath": "work/runs/run-20260907-001/shots/s01/clip.mp4",
    "voicePath": "work/runs/run-20260907-001/shots/s01/voice.mp3",
    "subtitle": "Why did the phone light up by itself"
  },
  { "shotId": "s02", "startMs": 6000, "endMs": 11000, "...": "..." }
]

Remember one rule: this table is the single truth. How long the cut runs, when a subtitle appears, which second the cover is grabbed from — all of it is computed from this table, and no hand-typed time value is allowed anywhere in the code. When a reader reports "the third shot's subtitle is half a second late," you do not go digging through a dozen ffmpeg commands for that 0.5; you look at how the third record's startMs was computed. A bug has exactly one place to hide.

The first thing computed from the table is the subtitle file. SRT's format is almost austere: an index, a line of start and end timecode, one or two lines of text, entries separated by a blank line.

srt.js
// The timecode is HH:MM:SS,mmm - a comma before the milliseconds, not a period.
// Write a period and most players simply behave as if there were no subtitles, silently.
function srtTime(ms) {
  const p = (n, w = 2) => String(n).padStart(w, '0')
  const h = Math.floor(ms / 3600000)
  const m = Math.floor(ms / 60000) % 60
  const s = Math.floor(ms / 1000) % 60
  return `${p(h)}:${p(m)}:${p(s)},${p(ms % 1000, 3)}`
}
 
export function renderSrt(entries, leadInMs = 200) {
  let index = 0
  return entries
    .filter((e) => e.subtitle)
    .map((e) => {
      index += 1
      // A little breathing room at both ends: no text on the frame the cut lands on,
      // and the cue clears a beat before the shot ends.
      const from = e.startMs + leadInMs
      const to = Math.max(from + 500, e.endMs - leadInMs)
      return `${index}\n${srtTime(from)} --> ${srtTime(to)}\n${e.subtitle}\n`
    })
    .join('\n')
}

Not one number in that code was invented on the spot; even the 200 breathing value is a parameter, changed once and applied everywhere. Its value is not cleverness, it is that "where does time come from" is now permanently settled.

So how do those pieces actually become one video? There are two completely different paths, and choosing wrong costs you either a tenfold slowdown or a cut that tears from the second shot onward.

Sign in and link WeChat to read

Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.

Mini program QR code

Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.

Interview questions

  • What are the risks of letting an LLM generate ffmpeg command lines directly, and how would you redesign it?如果让大模型直接生成 ffmpeg 命令来合成视频,会有什么风险?你会怎么改造这个设计?
    Common in ChinaCommon overseasDeep dive#prompt-injection#pipeline-design#reproducibility

    🔒 Unlock this chapter to see the answer keys.

  • An auto-generated episode comes out with audio and video out of sync. What is your debugging order, and why that order?一集自动生成的短剧成片出现音画不同步,你的排查顺序是什么?为什么是这个顺序?
    Common in ChinaCommon overseasIntermediate#debugging#av-sync#timeline

    🔒 Unlock this chapter to see the answer keys.

  • When can you concatenate video segments without re-encoding, and when must you re-encode?把多个视频片段拼成一条完整的视频,什么时候可以不重新编码,什么时候必须重编码?
    Common in ChinaCommon overseasBasic#ffmpeg#encoding#media-pipeline

    🔒 Unlock this chapter to see the answer keys.

Comments