Dayward AIWeChat Mini Program
Agent Evals and Observability in 7 Days: Turning "Seems Fine" Into Numbers You Can Reproduce
Week 1 · D3🔒 LockedAbout 4 hours

Let a Model Judge, Then Put the Judge Through an Exam

Anything code can decide should never go to a model, but open-ended quality genuinely has to. Today you write rubrics and pairwise comparison, measure the judge's three systematic biases with a position-swap experiment, and decide whether it is fit to use with an agreement coefficient rather than accuracy.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Decide whether a criterion belongs to a code, model or human grader, and state the concrete cost of choosing wrong
  2. Design a position-swap experiment that measures a judge's position bias, and explain why a verdict that flips has to be scored as a tie
  3. Explain why accuracy misleads when labels are heavily imbalanced, and use an agreement coefficient instead to decide whether a judge qualifies

Plain-Language Walkthrough

Two judges, and whether they agree with each other

A competition has two judges on the panel. What do you most want to know about them?

Most people say: are they judging correctly. That question has no answer. If a reference key existed that could tell you who was right, you would not need judges at all — the whole reason judges exist is that this thing has no key.

So the question that can actually be asked, and that actually helps, is: do the two judges agree with each other?

Agreement is measurable. Show the same contestants to both judges and count how often they reach the same verdict. High agreement means the standard is executable and somebody else would get the same result. Low agreement means the score depends mostly on who holds the pen, and writing it into a report is writing down one person's impression.

Today points that logic at a model. You will have a model score an agent's replies — a model grader, commonly called an LLM as judge. Then, immediately, something more important: put that judge through an exam of its own, quantify the systematic bias it carries, and use agreement to decide whether it is fit for the job.

Sign in and link WeChat to read

Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.

Mini program QR code

Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.

Interview questions

  • When would you refuse to use a model as the judge and reach for code or a human instead? State your decision order.什么情况下你会拒绝用模型当裁判,改用代码或人工?说出你的判断顺序。
    Common in ChinaCommon overseasIntermediate#evaluation#graders#llm-as-judge

    🔒 Unlock this chapter to see the answer keys.

  • How would you prove your model judge has no position bias? Describe an experiment someone could actually run.你怎么证明自己的模型裁判没有位置偏好?请描述一个可执行的实验。
    Common in ChinaCommon overseasDeep dive#evaluation#llm-as-judge#bias

    🔒 Unlock this chapter to see the answer keys.

  • Why measure judge quality with an agreement coefficient rather than accuracy? Give a scenario where accuracy lies.为什么评估裁判质量要用一致性系数而不是准确率?举一个准确率会骗人的场景。
    Common in ChinaCommon overseasDeep dive#evaluation#metrics#kappa

    🔒 Unlock this chapter to see the answer keys.

  • In a pairwise comparison the two rounds disagree. How do you handle that data point, and why not just pick one at random?成对比较时两轮结论不一致,你会怎么处理这条数据?为什么不是随机取一个?
    Common in ChinaCommon overseasIntermediate#evaluation#llm-as-judge#pairwise

    🔒 Unlock this chapter to see the answer keys.

Comments