Dayward AIWeChat Mini Program
Agent Evals and Observability in 7 Days: Turning "Seems Fine" Into Numbers You Can Reproduce
Week 1 · D1🔒 LockedAbout 4 hours

Why You Cannot Ship an Agent on "I Tried It a Few Times"

Start with the three things that make agents harder to evaluate than single-turn systems, settle the five terms this course runs on, and then turn non-determinism into two numbers you can put in a report: the odds of succeeding at least once, and the odds of succeeding every single time.

Today's goals 0/3

Sign in to tick these off and save your progress.

Start with the three things that make agents harder to evaluate than single-turn systems, settle the five terms this course runs on, and then turn non-determinism into two numbers you can put in a report: the odds of succeeding at least once, and the odds of succeeding every single time.

Sign in and link WeChat to read

Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.

Mini program QR code

Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.

Interview questions

  • You run the same task five times and get three passes and two failures. How do you report this agent's quality to your manager?同一个任务跑五次,三次对两次错。你会怎么向老板汇报这个 Agent 的质量?
    Common in ChinaCommon overseasIntermediate#evaluation#metrics#non-determinism

    🔒 Unlock this chapter to see the answer keys.

  • Why should you evaluate an agent against the final state of the environment rather than what its last message says?为什么评估 Agent 时要看环境的最终状态,而不是看它最后一条回复说了什么?
    Common in ChinaCommon overseasBasic#evaluation#graders#outcome

    🔒 Unlock this chapter to see the answer keys.

  • When should you report pass@k, and when must you report pass^k? Give an example of each and state the cost.什么场景该用 pass 的 at k,什么场景必须用 pass 的 k 次方?各举一个例子并说明代价。
    Common in ChinaCommon overseasIntermediate#evaluation#metrics#pass-at-k

    🔒 Unlock this chapter to see the answer keys.

  • A newly written evaluation task has a 0% pass rate over one hundred trials. What is your first reaction?一个新写的评估任务,跑一百次通过率是零。你的第一反应是什么?
    Common in ChinaCommon overseasIntermediate#evaluation#debugging#task-quality

    🔒 Unlock this chapter to see the answer keys.

Comments