The Regression Gate: Blocking a Drop Before It Merges
An evaluation only prevents incidents once it runs in the pipeline. Today you freeze a baseline into a snapshot, set a threshold on a score that moves every run without either missing regressions or crying wolf daily, and solve the two problems that actually bite: what a run costs, and who decides whether a red build is real.
Today's Goals
- Freeze an evaluation result into a baseline snapshot, and say what besides the score has to be recorded for it to be reproducible
- Define a regression rule on a noisy pass rate, and explain why a single lower score is not enough to call it a regression
- Design a cost-aware pipeline strategy, stating which tasks run on every commit and which only before a release
Plain-Language Walkthrough
A badge reader at the door
A badge reader in an office lobby is a very stupid machine. It does not know who you are or how urgently you need to get upstairs. It does one thing: wrong badge, turnstile stays shut.
Its value comes precisely from being stupid. If it could be talked into an exception, it would stop being a door and become a suggestion.
Everything built over the first five days is a suggestion. Graders, benchmark sets, judges, trajectory scoring, cost panels — all of them produce a report, and a report has to be read by a person. People are busy, people forget, and people tell themselves an hour before a release that this one is probably fine.
Today does one thing: wire that report into the turnstile. It has to shut when quality drops, and shut hard enough that nobody can walk around it without leaving a mark.
In engineering terms, "hard" has exactly one meaning: the process exits with a non-zero status code. That is the only thing a pipeline reads. Your evaluation script can print as much red text as it likes; if it exits zero, the merge happens.
Interview questions
Evaluation scores fluctuate on every run. How do you set a gate threshold that neither misses regressions nor fires false alarms every day?评估分数每次都在抖,你怎么定一个既不漏报也不天天误报的门禁阈值?
Common in ChinaCommon overseasDeep dive#ci#regression#thresholds🔒 Unlock this chapter to see the answer keys.
Beyond the pass rate, what else goes into a baseline snapshot? What breaks if you leave something out?基线快照里除了通过率,你还会记录什么?少记了会出什么问题?
Common in ChinaCommon overseasIntermediate#ci#baseline#reproducibility🔒 Unlock this chapter to see the answer keys.
Running the full evaluation on every commit is too expensive. How would you tier it, and on what basis?每次提交都跑全量评估太贵,你会怎么分层?依据是什么?
Common in ChinaCommon overseasIntermediate#ci#cost#strategy🔒 Unlock this chapter to see the answer keys.
Your evaluation gate keeps firing, and the team has learned to skip it. How would you change the process?评估门禁红了,团队却越来越习惯直接跳过。你会怎么改这条流程?
Common in ChinaCommon overseasDeep dive#ci#process#culture🔒 Unlock this chapter to see the answer keys.
