Let a Model Judge, Then Put the Judge Through an Exam

Interview questions

• When would you refuse to use a model as the judge and reach for code or a human instead? State your decision order.什么情况下你会拒绝用模型当裁判，改用代码或人工？说出你的判断顺序。

  Common in ChinaCommon overseasIntermediate#evaluation#graders#llm-as-judge

  How to reason about it · think before answering

  1. This tests selection discipline, not model knowledge. Jumping straight to prompt design misses the point: the interviewer wants to hear when you would NOT use a model judge.
  2. There is one ordering rule: if code can decide it, never hand it to a model. Whenever the criterion reduces to a settled fact, use code. Whether a refund happened is one database row; whether a file exists is one stat call; whether a status code matches is one comparison. Deterministic, free, instant.
  3. Only genuinely open-ended criteria reach the model tier: was it explained clearly, is the tone right, did it invent material that was not in the source. There is no fact to look up, so code cannot decide it.
  4. Human grading is the last tier, and it is not for running the full suite. Its one correct use is judging the judge: a few dozen human labels calibrate the model judge, and the model judge then runs at scale. Doing it the other way round spends the most expensive resource on the cases that need judgment least.
  5. State the cost in both directions - that is where candidates separate. Using a model where code would do buys you a noisy verdict that well-phrased lies can pass: an agent that calls no tool and merely says 'your refund has been processed' can score highly. Using code where a model is needed forces you to rewrite the criterion as keyword matching, so the grader starts measuring wording instead of quality, and the agent gets optimized toward hitting keywords nobody can read.
  6. Expected follow-up: what if half of an open-ended output is machine-checkable? Layer it rather than choosing. Whatever reduces to state goes to code (do the cited links resolve, are the required points present) and only the subjective remainder goes to the model. Attach both graders to the same task, each owning its half.

  分析过程 · 先想清楚再作答

  1. 这题考的是选型纪律，不是模型知识。上来就聊提示词怎么写的，方向已经偏了——面试官想听的是你在什么时候**不**用它。
  2. 判断顺序只有一条：**能用代码判的绝不交给模型**。判据只要能落到一个确定的事实上，就该用代码。退款有没有发生是一行数据库记录，文件有没有生成是一次 stat，接口返回码对不对是一次比较，这些一律代码判——确定、免费、瞬时。
  3. 只有当判据本身是开放式的，才轮到模型：讲清楚了没有、语气合不合适、有没有编造材料里没有的内容。这类判据没有可查的事实，代码判不了。
  4. 人工型是最后一档，而且**不是拿来跑全量的**。它的正确用法只有一个：当裁判的裁判。用几十条人工标注去校准模型裁判，然后让模型裁判去跑全量。反过来做就是把最贵的资源浪费在最不需要判断力的地方。
  5. 两个方向的代价要分别说清楚，这是区分度所在。**该用代码却用了模型**：你花钱买了一个会抖、会被漂亮话骗过去的判定——一个什么工具都不调、只会说「已为您办理退款」的 Agent，在模型裁判下可能拿高分。**该用模型却用了代码**：为了让代码判得了，你会把判据改成关键词匹配，于是评分器考的是措辞而不是质量，Agent 会朝这个方向被优化，最后你得到一个句句踩关键词、没人看得懂的系统。
  6. 可预期的追问是「开放式输出里有一半能用代码判怎么办」。答案是**分层而不是二选一**：能落到结果态的部分交给代码（引用的链接解不解析得开、必须覆盖的要点在不在），剩下真正主观的部分才交给模型。同一个任务上挂两个评分器，各管各的那一半。

  Key points

  • The order is fixed: never give a model what code can decide; humans only calibrate the other two.
  • If the criterion reduces to a settled fact, use code: outcome state, fields, status codes - deterministic and free.
  • Model where code would do: a noisy, paid verdict that an agent producing only nice words can pass.
  • Code where a model is needed: the criterion degrades into keyword matching and the grader measures wording, not quality.
  • Layer open-ended output instead of choosing: checkable parts to code, subjective remainder to the model.

  答题要点

  • 顺序是固定的：能用代码判的绝不交给模型，人工只用来校准前两者。
  • 判据能落到确定事实上就用代码：结果态、字段、返回码，确定且免费。
  • 该用代码却用模型：判定会抖、会花钱、会被不干活只说漂亮话的 Agent 骗过。
  • 该用模型却用代码：判据被迫退化成关键词匹配，评分器开始考措辞而不是质量。
  • 开放式输出要分层，不是二选一：可查的部分给代码，主观的剩余部分给模型。
• How would you prove your model judge has no position bias? Describe an experiment someone could actually run.你怎么证明自己的模型裁判没有位置偏好？请描述一个可执行的实验。

  Common in ChinaCommon overseasDeep dive#evaluation#llm-as-judge#bias

  How to reason about it · think before answering

  1. The question asks for a runnable experiment, so 'I would watch out for ordering effects' scores zero. Specify the data, the judging runs, the statistics, and the decision rule.
  2. Design: take a set of answer pairs and judge each pair twice, once with A first and once with B first, everything else identical. One pair yields two verdicts; N pairs yield 2N.
  3. The core statistic is the front-slot win rate: among all decisive single rounds, the fraction where the winner happened to be presented first. With no position bias it should sit at 50%. Significantly above is position bias, and significantly below is too - some models favor the second option, which is equally systematic.
  4. The second statistic is the flip rate: the fraction of pairs whose two rounds contradict each other. It is not a detector but a loss estimate - a 40% flip rate means that judging once leaves 40% of your conclusions determined by position, and the report will not say which ones.
  5. Sampling matters: deliberately include evenly matched pairs. Position bias only shows up when quality is close; a set of lopsided pairs buries the bias under the quality gap and you will measure a comforting 50%. This is the easiest place for the experiment to lie to you.
  6. Close with the decision rule and the control. Use a statistical test rather than eyeballing: a binomial test of the front-slot rate against 50%, noting that a small sample gives an interval too wide to conclude anything. For the control, run the same data through a judge known to be unbiased and confirm the harness reports 50% - without that step you cannot distinguish 'measured a real bias' from 'the harness always reports bias'.
  7. Expected follow-up: what do you do once bias is confirmed? In order: switch to independent per-item scoring to avoid pairwise entirely; if pairwise is required, run the swap and call flips ties; only then consider changing the judge model.

  分析过程 · 先想清楚再作答

  1. 这题问的是「可执行的实验」，所以答「我会注意顺序的影响」是零分。必须给出数据怎么造、判决怎么跑、统计什么数字、判据是什么。
  2. 实验设计：取一批成对的答案，每一对**正反各评一次**。第一轮把 A 放前面，第二轮把 B 放前面，其余条件完全相同。一对答案得到两条判决，N 对得到 2N 条。
  3. 要统计的核心数字是**前排胜率**：在所有分出胜负的单轮判决里，赢家恰好被摆在前面的比例。没有位置偏好时它应该是 50%。显著高于 50% 就是位置偏好，显著低于 50% 也是（少数模型会偏向后一个，同样是系统性偏差）。
  4. 第二个数字是**结论翻转率**：两轮结论互相矛盾的对数占比。它的用途不是检测，而是估算损失——翻转率是 40% 就意味着，只评一轮的话你的结论里有四成是位置决定的，而报告不会告诉你是哪四成。
  5. 样本怎么造是有讲究的：**要故意包含一批势均力敌的答案对**。位置偏好只在质量接近时发作，全拿一优一劣的对子去测，偏差被质量差距盖住，你会测出一个漂亮的 50% 并得出错误结论。这也是这个实验最容易做假的地方。
  6. 最后要说判据与对照。判据用统计检验而不是眼看：前排胜率对 50% 做二项检验，样本量不够时区间会宽到什么都说明不了。对照组的做法是拿一个**已知无偏**的裁判跑同一批数据，确认实验代码本身会输出 50%——不做这一步，你无法区分「量出了偏差」和「实验代码恒定报告有偏差」。
  7. 可预期的追问是「测出来有偏差之后怎么办」。顺序是：先换成逐条独立打分绕开成对比较；必须成对时用交换实验并把翻转的判平；两者都不行才考虑换裁判模型。

  Key points

  • Judge each pair twice with the order swapped; N pairs give 2N verdicts under otherwise identical conditions.
  • Primary metric: front-slot win rate, which should be 50% for an unbiased judge.
  • Secondary metric: flip rate, estimating how many single-round conclusions position would have decided.
  • Deliberately include evenly matched pairs, or the quality gap masks the bias.
  • Decide with a binomial test, and run a known-unbiased judge as a control to confirm the harness reports 50%.

  答题要点

  • 同一对答案正反各评一次，N 对得到 2N 条判决，其余条件完全相同。
  • 主指标是前排胜率：赢家恰好排在前面的比例，无偏时应为 50%。
  • 辅指标是翻转率，用来估算「只评一轮」会让多少结论由位置决定。
  • 样本必须故意包含势均力敌的对子，否则质量差距会把偏差盖住。
  • 判据用二项检验，并拿一个已知无偏的裁判做对照，确认实验本身会输出 50%。
• Why measure judge quality with an agreement coefficient rather than accuracy? Give a scenario where accuracy lies.为什么评估裁判质量要用一致性系数而不是准确率？举一个准确率会骗人的场景。

  Common in ChinaCommon overseasDeep dive#evaluation#metrics#kappa

  How to reason about it · think before answering

  1. The discriminator is not whether you know kappa, but whether you can state the conditions and the magnitude by which accuracy lies. 'Accuracy is unreliable on imbalanced data' is common knowledge, not an answer.
  2. Lead with a concrete scenario. You send thirty samples for human labeling; humans mark twenty-seven acceptable and three not. That ratio is realistic - most production samples sent for labeling are fine. Now compare a judge that looks at nothing and always says pass: it is right twenty-seven times, for 90% accuracy. A zero-information constant scores 90%.
  3. And this is not a contrived extreme. The more imbalanced the labels, the higher it goes: on ninety-nine to one data the same constant reports 99%. On imbalanced data accuracy is systematically optimistic, not occasionally wrong.
  4. Kappa applies one correction: compute pe, the agreement two labelers would reach by chance given their own label distributions, and subtract it from the observed agreement po, as (po - pe) over (1 - pe). For the constant judge po is 0.9 and pe is also 0.9, so kappa is exactly 0. Kappa of zero means the judge is equivalent to guessing from the label distribution, regardless of how high its accuracy looks.
  5. Resolution makes the case even better. In a measured comparison, removing a judge's length bias moved accuracy from 86.7% to 93.3%, about six and a half points, while kappa jumped from 0.259 to 0.714 - from nearly useless to over the admission line. The same improvement, an order of magnitude apart in signal. And before the fix, the always-pass constant scored 90%, higher than the real judge. Gate on accuracy and you eliminate the only judge carrying information.
  6. Close with thresholds and edge cases. Rules of thumb: below 0.2 is nearly useless, 0.4 to 0.6 is marginal, 0.6 to 0.8 is pipeline-grade. Two edges are worth naming: when both labelers assign every sample to one class kappa is undefined and the implementation must return 0 rather than 1; and with more than two labelers you switch to Fleiss' kappa, same idea.

  分析过程 · 先想清楚再作答

  1. 这题的区分度不在「知不知道 kappa」，而在能不能把准确率骗人的**条件与幅度**说具体。只说「数据不均衡时准确率不可靠」，答的是常识。
  2. 先把场景摆出来，越具体越好：你送了三十条样本去人工标注，人工判定二十七条合格、三条不合格。这个比例接近真实——线上送去标注的样本本来就大多数是正常的。现在拿一个**什么都不看、一律判通过**的裁判来比：它判对了二十七条，准确率 90%。**一个零信息量的常量，拿到了 90% 的准确率。**
  3. 而且这不是极端构造。标签越不均衡，这个数字越高：九十九比一的数据上，同一个常量裁判能报出 99%。准确率在不均衡数据上是一个**系统性乐观**的指标，不是偶尔失灵。
  4. kappa 的修正只有一步：先算出「随机也能一致」的比例 pe，再从观察一致率 po 里扣掉它，公式是 (po - pe) 除以 (1 - pe)。那个常量裁判的 po 是 0.9，pe 也是 0.9，kappa 正好是 0。**kappa 为 0 的含义是「这个裁判等价于按标签分布瞎猜」，与它的准确率有多高完全无关。**
  5. 更能说明问题的是分辨率。实测过一组对照：同一个裁判修掉长度偏好之后，准确率从 86.7% 升到 93.3%、涨了 6.6 个点，而 kappa 从 0.259 跳到 0.714——从「几乎没用」直接跨过准入线。同一个改进，两个指标的反应差了一个数量级。而在修之前，那个一律判通过的常量裁判准确率是 90%，**比真裁判还高**。用准确率做准入，你会淘汰掉唯一一个带信息的裁判。
  6. 最后给阈值与边界：经验上 0.2 以下几乎没用，0.4 到 0.6 勉强，0.6 到 0.8 可以进流水线。两个边界情形要提：两人都把所有样本判成同一类时 kappa 无定义，实现里必须返回 0 而不是 1；标注者多于两人时换 Fleiss' kappa，思路一样。

  Key points

  • Scenario: with twenty-seven of thirty acceptable, an always-pass constant already scores 90% accuracy.
  • The more imbalanced the labels, the higher it climbs - accuracy is systematically optimistic here.
  • Kappa subtracts pe, the chance agreement, and the constant judge lands at exactly 0.
  • Kappa resolves far better: one fix moved accuracy 6.6 points but kappa from 0.259 to 0.714.
  • 0.6 and up is pipeline-grade; kappa is undefined when both labelers pick one class and must return 0; use Fleiss for more labelers.

  答题要点

  • 场景：三十条里二十七条合格时，一律判通过的常量裁判准确率就有 90%。
  • 标签越不均衡这个数字越高，所以准确率是系统性乐观，不是偶尔失灵。
  • kappa 把「随机也能一致」的 pe 扣掉，那个常量裁判的 kappa 正好是 0。
  • kappa 的分辨率高得多：同一次修复准确率涨 6.6 个点，kappa 从 0.259 到 0.714。
  • 阈值 0.6 起可进流水线；两人同判一类时 kappa 无定义必须返回 0，多人换 Fleiss。
• In a pairwise comparison the two rounds disagree. How do you handle that data point, and why not just pick one at random?成对比较时两轮结论不一致，你会怎么处理这条数据？为什么不是随机取一个？

  Common in ChinaCommon overseasIntermediate#evaluation#llm-as-judge#pairwise

  How to reason about it · think before answering

  1. This looks like a detail but tests whether you will write noise into a report as if it were signal. Answering 'take the first round because it matches real usage order' has already fallen in.
  2. The correct handling is to call it a tie. A flip means precisely one thing: that conclusion was decided by position, not by quality. Its information content is zero, and a tie is the honest way to say 'these two cannot be separated'. A tie is not a failure to conclude; it is a conclusion.
  3. Why not pick at random: that writes noise into the report, and half the time it will coincidentally match your expectation. The coincidental half increases your confidence in the judge, so the error gets locked in. The cost of treating noise as signal is not one missing data point but one fabricated one.
  4. Why not always take the first round either: that swallows the position bias wholesale. The entire purpose of the swap was to remove positional influence, so resolving by position undoes the experiment.
  5. Also specify what happens downstream. A tie is neither a pass nor a fail; it belongs in its own bucket in the report, alongside the count of decisive conclusions. A high tie rate is itself a finding - it says your rubric cannot separate these answers, either because it is too coarse or because the two systems really are comparable.
  6. Expected follow-up: what if ties leave too few data points? Fix it on the data side - more samples, or finer criteria so the rubric can discriminate - not by relaxing the merge rule. Relaxing it trades conclusion quality for conclusion count, which is backwards.

  分析过程 · 先想清楚再作答

  1. 这题看着是个细节，实际上考的是「会不会把噪声当成信号写进报告」。答「取第一轮的结果，因为它更接近真实使用顺序」的，已经掉进坑里了。
  2. 正确处理是**判平局**。翻转的含义很明确：这条结论是由位置决定的，不是由质量决定的。它携带的信息量是零，而平局恰恰是「分不出来」的如实表达。平局不是认输，是一个有内容的结论。
  3. 为什么不能随机取一个：随机取等于把噪声写进了报告，而且它有一半的概率碰巧和你的预期一致。碰巧一致的那一半会让你更相信这个裁判，于是错误被固化。**把噪声当成信号的代价，不是少了一条数据，而是多了一条假数据。**
  4. 为什么也不能固定取第一轮：那等于把位置偏好整个吞了下去。你测这个实验的全部目的就是消除位置的影响，最后却按位置取结论，等于没做。
  5. 顺带要说清楚平局的下游处理。平局不能当成失败，也不能当成通过，它应该单独成一档进报告：多少条有效结论、多少条判平。平局比例过高本身就是一个结论——它说明这批答案在你的量表下分不出差别，要么量表太粗，要么这两个系统确实旗鼓相当。
  6. 可预期的追问是「平局太多导致样本量不够怎么办」。答案不是放宽合并规则，而是从数据侧解决：加样本量、或者把判据拆细让量表分得动。放宽合并规则等于用降低结论质量来换结论数量，方向反了。

  Key points

  • Call every flipped pair a tie: it was decided by position, not quality, and carries zero information.
  • Picking at random writes noise into the report and half the time appears to confirm your expectation, locking in the error.
  • Always taking the first round swallows the position bias and wastes the swap entirely.
  • Report ties as their own bucket - neither a pass nor a fail.
  • Too many ties is a data-side problem: more samples or finer criteria, not a looser merge rule.

  答题要点

  • 翻转的结论一律判平：它由位置而不是由质量决定，信息量是零。
  • 随机取一个是把噪声写进报告，还会有一半概率碰巧印证你的预期，固化错误。
  • 固定取第一轮等于吞掉位置偏好，交换实验白做。
  • 平局单独成一档进报告，不能算失败也不能算通过。
  • 平局过多要从数据侧解决：加样本、拆细判据，而不是放宽合并规则。

Comments