Dayward AIWeChat Mini Program
← All course question banks

Agent Evals and Observability in 7 Days: Turning "Seems Fine" Into Numbers You Can Reproduce

Course questions · 28 / 28

D1 Why You Cannot Ship an Agent on "I Tried It a Few Times"

  • You run the same task five times and get three passes and two failures. How do you report this agent's quality to your manager?同一个任务跑五次,三次对两次错。你会怎么向老板汇报这个 Agent 的质量?
    Common in ChinaCommon overseasIntermediate#evaluation#metrics#non-determinism

    🔒 Unlock D1 to see the answer keys.

  • Why should you evaluate an agent against the final state of the environment rather than what its last message says?为什么评估 Agent 时要看环境的最终状态,而不是看它最后一条回复说了什么?
    Common in ChinaCommon overseasBasic#evaluation#graders#outcome

    🔒 Unlock D1 to see the answer keys.

  • When should you report pass@k, and when must you report pass^k? Give an example of each and state the cost.什么场景该用 pass 的 at k,什么场景必须用 pass 的 k 次方?各举一个例子并说明代价。
    Common in ChinaCommon overseasIntermediate#evaluation#metrics#pass-at-k

    🔒 Unlock D1 to see the answer keys.

  • A newly written evaluation task has a 0% pass rate over one hundred trials. What is your first reaction?一个新写的评估任务,跑一百次通过率是零。你的第一反应是什么?
    Common in ChinaCommon overseasIntermediate#evaluation#debugging#task-quality

    🔒 Unlock D1 to see the answer keys.

D2 Benchmark Sets: Turning the Incidents You Already Had Into Reproducible Tasks

  • Your manager gives you two days to build an evaluation for an agent that is already in production. Where do you get the test tasks?老板让你两天内给一个已上线的 Agent 建评估,你会从哪里找测试任务?
    Common in ChinaCommon overseasIntermediate#evaluation#golden-set#task-design

    🔒 Unlock D2 to see the answer keys.

  • What is one-sided optimization? Give an example of where a suite containing only positive cases drives a system.什么叫单向优化?举例说明只写正向用例会把系统带到什么地方。
    Common in ChinaCommon overseasIntermediate#evaluation#golden-set#negative-tests

    🔒 Unlock D2 to see the answer keys.

  • What makes an evaluation task well-formed, and how do you verify the task itself is not broken?一条评估任务应该满足什么条件才算合格?你怎么验证它本身没写错?
    Common in ChinaCommon overseasDeep dive#evaluation#task-quality#reference-solution

    🔒 Unlock D2 to see the answer keys.

  • What pass rates do you expect from a capability suite versus a regression suite, and why are they different?能力评估和回归评估的通过率,你分别期望它们是多少?为什么不一样?
    Common in ChinaCommon overseasIntermediate#evaluation#capability-vs-regression#reporting

    🔒 Unlock D2 to see the answer keys.

D3 Let a Model Judge, Then Put the Judge Through an Exam

  • When would you refuse to use a model as the judge and reach for code or a human instead? State your decision order.什么情况下你会拒绝用模型当裁判,改用代码或人工?说出你的判断顺序。
    Common in ChinaCommon overseasIntermediate#evaluation#graders#llm-as-judge

    🔒 Unlock D3 to see the answer keys.

  • How would you prove your model judge has no position bias? Describe an experiment someone could actually run.你怎么证明自己的模型裁判没有位置偏好?请描述一个可执行的实验。
    Common in ChinaCommon overseasDeep dive#evaluation#llm-as-judge#bias

    🔒 Unlock D3 to see the answer keys.

  • Why measure judge quality with an agreement coefficient rather than accuracy? Give a scenario where accuracy lies.为什么评估裁判质量要用一致性系数而不是准确率?举一个准确率会骗人的场景。
    Common in ChinaCommon overseasDeep dive#evaluation#metrics#kappa

    🔒 Unlock D3 to see the answer keys.

  • In a pairwise comparison the two rounds disagree. How do you handle that data point, and why not just pick one at random?成对比较时两轮结论不一致,你会怎么处理这条数据?为什么不是随机取一个?
    Common in ChinaCommon overseasIntermediate#evaluation#llm-as-judge#pairwise

    🔒 Unlock D3 to see the answer keys.

D4 Trajectory Evaluation: It Reached the Destination, but How Many Things Did It Hit

  • An agent ships with all outcomes correct, but its average cost has gone up tenfold. How would your evaluation system catch that?一个 Agent 上线后结果全对,但平均成本翻了十倍。你的评估体系怎么才能发现这件事?
    Common in ChinaCommon overseasIntermediate#evaluation#trajectory#cost

    🔒 Unlock D4 to see the answer keys.

  • Would you assert that an agent's tool-call sequence must exactly match a reference workflow? Give your reasoning.你会把「工具调用顺序必须与参考流程完全一致」写进评分器吗?说出你的理由。
    Common in ChinaCommon overseasIntermediate#evaluation#trajectory#graders

    🔒 Unlock D4 to see the answer keys.

  • How do you automatically detect that an agent is stuck in a loop? Give your rule and its false-positive risks.怎么自动判定一个 Agent 陷入了死循环?给出你的规则,以及它的误判风险。
    Common in ChinaCommon overseasIntermediate#evaluation#trajectory#loop-detection

    🔒 Unlock D4 to see the answer keys.

  • How do you automate evaluation of multi-turn conversations, and what problems does a simulated user introduce?多轮对话场景怎么做自动化评估?模拟用户会带来什么问题?
    Common in ChinaCommon overseasDeep dive#evaluation#multi-turn#simulated-user

    🔒 Unlock D4 to see the answer keys.

D5 Observability: Putting a Flight Recorder on Every Run

  • Why instrument an agent with a public semantic convention instead of a field-name scheme your team invents?为什么给 Agent 埋点要用公开的语义约定,而不是团队自己定一套字段名?
    Common in ChinaCommon overseasBasic#observability#opentelemetry#semantic-conventions

    🔒 Unlock D5 to see the answer keys.

  • Would you adopt a standard that is still in development and whose attribute names may change? How do you control the upgrade risk?一个仍在开发中、属性名还会改的标准,你会现在就用吗?怎么控制升级风险?
    Common in ChinaCommon overseasDeep dive#observability#opentelemetry#dependency-risk

    🔒 Unlock D5 to see the answer keys.

  • How would you layer the trace tree for one agent run, and what attributes go on each layer?一次 Agent 运行的链路树你会怎么分层?每层各记哪些属性?
    Common in ChinaCommon overseasIntermediate#observability#tracing#span-design

    🔒 Unlock D5 to see the answer keys.

  • Recording every trace in production is too expensive. How do you design sampling so that you do not throw away the failures you actually need?线上全量记录太贵,你怎么设计采样策略才不会把真正的故障样本丢掉?
    Common in ChinaCommon overseasDeep dive#observability#sampling#production

    🔒 Unlock D5 to see the answer keys.

D6 The Regression Gate: Blocking a Drop Before It Merges

  • Evaluation scores fluctuate on every run. How do you set a gate threshold that neither misses regressions nor fires false alarms every day?评估分数每次都在抖,你怎么定一个既不漏报也不天天误报的门禁阈值?
    Common in ChinaCommon overseasDeep dive#ci#regression#thresholds

    🔒 Unlock D6 to see the answer keys.

  • Beyond the pass rate, what else goes into a baseline snapshot? What breaks if you leave something out?基线快照里除了通过率,你还会记录什么?少记了会出什么问题?
    Common in ChinaCommon overseasIntermediate#ci#baseline#reproducibility

    🔒 Unlock D6 to see the answer keys.

  • Running the full evaluation on every commit is too expensive. How would you tier it, and on what basis?每次提交都跑全量评估太贵,你会怎么分层?依据是什么?
    Common in ChinaCommon overseasIntermediate#ci#cost#strategy

    🔒 Unlock D6 to see the answer keys.

  • Your evaluation gate keeps firing, and the team has learned to skip it. How would you change the process?评估门禁红了,团队却越来越习惯直接跳过。你会怎么改这条流程?
    Common in ChinaCommon overseasDeep dive#ci#process#culture

    🔒 Unlock D6 to see the answer keys.

D7 Examining the Suite Itself: Saturation, Broken Tasks and Grading Defects

  • Your evaluation suite is passing at 98 percent. What do you do next?你们的评估套件通过率已经百分之九十八了,接下来你会做什么?
    Common in ChinaCommon overseasIntermediate#evaluation#saturation#suite-health

    🔒 Unlock D7 to see the answer keys.

  • A model suddenly drops a lot of points on your suite. How do you tell a real regression from a broken grader?一个模型在你的套件上突然掉了很多分,你怎么区分是模型退化还是判分写错了?
    Common in ChinaCommon overseasDeep dive#evaluation#debugging#grading-defect

    🔒 Unlock D7 to see the answer keys.

  • Why should you not trust an evaluation score before someone has read transcripts? And how do you make transcript reading a team habit?为什么说没读过轨迹就不该相信评估分数?你会怎么把读轨迹变成团队习惯?
    Common in ChinaCommon overseasIntermediate#evaluation#transcripts#process

    🔒 Unlock D7 to see the answer keys.

  • Build your own evaluation framework or adopt an existing one? Give your decision criteria, and say what you check in the license.自建评估框架还是用现成的?说出你的判断依据,以及选型时会看许可的哪些方面。
    Common in ChinaCommon overseasDeep dive#evaluation#tooling#licensing

    🔒 Unlock D7 to see the answer keys.