Voiceover, Subtitles, and Audio Tracks: Multi-Character Voices, Timeline Alignment, and Subtitle Files
Give every character their own voice, use voice duration to correct shot duration in return, generate subtitle files with a correct timeline, and get the relationship between background music and dialogue right.
Today's Goals
- Assign a voice per character and batch-synthesize dialogue, producing audio files ready to use directly
- Align audio duration with shot duration, and generate subtitle files with a correct timeline
- Explain the parameters in speech synthesis that most affect the finished feel, and the boundaries for handling background music
Yesterday you got a batch of moving shots. Today you give them voices — and settle a question the first four days kept deferring: how long should this shot actually be? When you finish, scroll back up and tick the three goals off.
Plain-Language Walkthrough
Break one default assumption first, or today's whole chapter reads as the only option available.
Not every video model produces a silent film. Some return picture only, with an empty audio track; some include ambient sound; some will generate lip-sync along with spoken lines written into the prompt, and the official docs show exactly that form, sometimes allowing a reference audio clip to fix the timbre. So "once you have the shot you must dub it" depends on which model you used — before you start, check whether the output even has an audio stream.
If the model can speak for itself, why does this course still run a separate synthesis step? Four engineering differences:
- The lines are controllable word for word. The script's dialogue is fixed. Let the model improvise and it will never say exactly that line — and in drama, one wrong word is a wrong line.
- The timbre stays stable across shots. A character appears in six shots; if each clip generates its own voice, one person ends up with six voices. This is the audio version of the face-swap problem, and audiences are more sensitive to voice than to face.
- The timeline becomes computable. The whole pipeline's timeline is derived backwards from voice duration (that is the second half of today). With speech baked into the video you cannot measure when each line starts or how long it runs — subtitle timecodes cannot be generated, and music ducking cannot be aligned.
- Changing one word differs by four orders of magnitude. Re-synthesizing a line costs a fraction of a cent; re-generating a shot costs dollars.
Conversely, if your piece explicitly wants no subtitles, the lines need not be exact, and lip-sync matters more, letting the video model speak is the easier choice — its lip movement is generated with the picture, and post cannot match that. Today teaches the controllable path; knowing the other one exists is how you know what you are choosing.
One voice per character
In a recording booth the voice director's first job is not to roll, it is casting: who voices the lead, who voices the second female. Once decided, it goes on the wall chart and does not change all season. Audiences remember voices better than you expect — change a character's voice in episode two and the comments immediately ask who this is.
So step one of dubbing is an assignment table, and that table should not be improvised on dubbing day. It belongs to the character card defined on D2: each character has a voiceId field holding the vendor's system voice identifier. Cross-episode consistency comes from that record, not from a temporary map inside the dubbing code — which is why this course calls the character card a record rather than a parameter.
MiniMax's system voices include a set tuned for short drama, named after the archetype: badao_shaoye the domineering heir, junlang_nanyou the handsome boyfriend, lengdan_xiongzhang the aloof senior, chunzhen_xuedi the innocent junior, bingjiao_didi the possessive little brother; on the female side wumei_yujie, danya_xuejie, tianxin_xiaoling, qiaopi_mengmei, diadia_xuemei. There is also a general set such as male-qn-jingying and female-chengshu, good for narration and background characters.
Beyond the voice itself, three parameters directly shape the finished feel:
Emotion (emotion) is the most overlooked and the most audible. Its values are a fixed enumeration: happy, sad, angry, fearful, disgusted, surprised, calm, fluent, whisper. The same line — "that phone is not yours" — is a cool warning under calm and a confrontation under angry; two completely different scenes. The engineering move is to give each character a baseline emotion and let individual lines override it. Short-form drama's emotional turns live in those overrides.
Speed (speed) ranges from 0.5 to 2, and it has a hidden effect: speed changes audio duration, and duration propagates into the shot. So speed cannot be nudged casually at this step; nudge it and you re-align the timeline.
Volume and pitch (vol, pitch) usually stay at their defaults. To make a character sound younger or heavier, change the voice, do not pull the pitch — pitched audio has an obvious synthetic edge.
Finally one return-value detail you must know: data.audio is a hex string by default, not base64. In Node that is Buffer.from(json.data.audio, "hex"). Decoding it as base64 gives you a file that saves fine and will not play, and the error message points nowhere near the cause.
async function synthesize({ text, voiceId, emotion, speed, outPath }) {
const res = await fetch(`${process.env.MINIMAX_BASE_URL}/v1/t2a_v2`, {
method: 'POST',
headers: {
Authorization: `Bearer ${process.env.MINIMAX_API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: 'speech-2.8-hd',
text,
stream: false,
voice_setting: { voice_id: voiceId, speed, vol: 1, pitch: 0, emotion },
audio_setting: { sample_rate: 32000, bitrate: 128000, format: 'mp3', channel: 1 },
}),
})
const json = await res.json()
if (json.base_resp?.status_code) throw new Error(`synthesis failed ${json.base_resp.status_code}`)
// data.audio is a hex string by default, not base64 - decoding as base64 gives an unplayable file
await writeFile(outPath, Buffer.from(json.data.audio, 'hex'))
// audio_length is in milliseconds and goes straight into the timeline
return { path: outPath, durationMs: json.extra_info.audio_length }
}import os
import httpx
def synthesize(text: str, voice_id: str, emotion: str, speed: float, out_path: str) -> dict:
res = httpx.post(
f"{os.environ['MINIMAX_BASE_URL']}/v1/t2a_v2",
headers={"Authorization": f"Bearer {os.environ['MINIMAX_API_KEY']}"},
json={
"model": "speech-2.8-hd",
"text": text,
"stream": False,
"voice_setting": {"voice_id": voice_id, "speed": speed, "vol": 1, "pitch": 0, "emotion": emotion},
"audio_setting": {"sample_rate": 32000, "bitrate": 128000, "format": "mp3", "channel": 1},
},
timeout=120,
)
data = res.json()
if data.get("base_resp", {}).get("status_code"):
raise RuntimeError(f"synthesis failed {data['base_resp']['status_code']}")
# data.audio is a hex string by default, not base64 - base64 gives an unplayable file
with open(out_path, "wb") as f:
f.write(bytes.fromhex(data["data"]["audio"]))
# audio_length is in milliseconds and goes straight into the timeline
return {"path": out_path, "duration_ms": data["extra_info"]["audio_length"]}Voices cast, parameters tuned, audio on disk — and now the real problem appears: this clip runs 3.3 seconds, but the shot you generated yesterday is 3 seconds. Who gives way?
Interview questions
When the synthesized speech and the shot duration disagree, which side do you adjust, and why?语音时长和画面时长对不上,你会调哪一边?为什么?
Common in ChinaCommon overseasIntermediate#timeline#tts#pipeline-design🔒 Unlock this chapter to see the answer keys.
Where do you get subtitle timestamps from, and what do you do when the API does not provide them?字幕的时间戳你会怎么拿?接口不给时间戳时有什么替代方案?
Common in ChinaCommon overseasIntermediate#subtitles#timeline🔒 Unlock this chapter to see the answer keys.
In a multi-character pipeline, how do you guarantee the same character keeps the same voice across episodes?多角色配音里,怎么保证同一个角色跨集用的是同一个声音?
Common in ChinaCommon overseasBasic#tts#consistency#provider-abstraction🔒 Unlock this chapter to see the answer keys.
