Concurrency and Quotas: Starting Multiple Episodes at Once Without Blowing Through Any Provider's Limits
Produce multiple episodes in parallel, use a queue, a token bucket, and priorities to control the load on each provider, and handle the queuing, starvation, and placeholder problems long tasks hit under concurrency.
Today's Goals
- Run multiple episodes in parallel with a queue, and cap concurrency and rate separately per provider
- Prioritize tasks so an urgent rush job doesn't starve a queued long-running task
- Make the right backoff and degradation decisions among over-limit, rate-limited, and queue-timeout conditions
Yesterday's engine runs one episode reliably and resumes from a checkpoint. Today we push it from one episode at a time to five episodes at once, and you immediately hit a new problem: machines are plentiful, quota is not. When you finish, scroll back up and tick the three goals off.
Plain-Language Walkthrough
Three units at once and one crane
When a production scales up, the line producer's first thought is "run several units in parallel." The second thought usually arrives within half a day: three units all want the camera crane, and there is one crane. You can hire more people and rent more lights, but that crane is one crane.
On a generative pipeline the crane is provider quota. Your process can open a hundred connections and your cloud host can grow to sixteen cores, but the rate ceiling MiniMax gives you does not move. Those numbers are public; we copy them from the official rate-limit page, at the topped-up tier:
| API | Topped-up RPM |
|---|---|
| Speech T2A v2 | 20 |
| Image generation | 10 |
| Video generation v1 | 20 |
| Text MiniMax-M3 | 200 |
Convert those into units you feel. One episode has forty storyboard entries, each with at least one line, so forty syntheses. Speech is twenty per minute, so dubbing alone queues for two minutes — for one episode. Five in parallel is two hundred calls, ten minutes. First frames are tighter: images are ten per minute, so five episodes' two hundred first frames take twenty minutes, twice as slow as dubbing.
Those two numbers overturn a natural intuition. Most people assume the parallel bottleneck is "video generation is slow" and pour effort into optimizing video; but computed against quota, the stage that jams the whole line is often the one that looks cheapest and fastest. An image is two and a half cents and returns in seconds, and you may only call it ten times a minute.
So today's job is not "raise the concurrency setting" but to install a gate for each provider's each API class, so your process holds itself inside the quota. That sounds like self-restraint, and it is the opposite: only by limiting yourself deliberately can you run continuously at the top edge of the quota. The unlimited version overshoots, gets refused, backs off, retries, overshoots again — and averages lower throughput.
Interview questions
Many jobs share one vendor's quota. How would you design the rate limiting?多个任务共用一家厂商的额度,你会怎么设计限流?
Common in ChinaCommon overseasIntermediate#rate-limiting#concurrency#scheduling🔒 Unlock this chapter to see the answer keys.
Beyond backing off and retrying, what else should happen when you get rate limited?收到限流响应之后,除了退避重试还该做什么?
Common in ChinaCommon overseasDeep dive#rate-limiting#error-handling#retry🔒 Unlock this chapter to see the answer keys.
Priority queues starve low-priority work. How do you prevent that?优先级队列容易出现饿死,你会怎么防?
Common in ChinaCommon overseasBasic#scheduling#priority-queue#fairness🔒 Unlock this chapter to see the answer keys.
