Why You Cannot Ship an Agent on "I Tried It a Few Times"
Start with the three things that make agents harder to evaluate than single-turn systems, settle the five terms this course runs on, and then turn non-determinism into two numbers you can put in a report: the odds of succeeding at least once, and the odds of succeeding every single time.
Start with the three things that make agents harder to evaluate than single-turn systems, settle the five terms this course runs on, and then turn non-determinism into two numbers you can put in a report: the odds of succeeding at least once, and the odds of succeeding every single time.
Interview questions
You run the same task five times and get three passes and two failures. How do you report this agent's quality to your manager?同一个任务跑五次,三次对两次错。你会怎么向老板汇报这个 Agent 的质量?
Common in ChinaCommon overseasIntermediate#evaluation#metrics#non-determinism🔒 Unlock this chapter to see the answer keys.
Why should you evaluate an agent against the final state of the environment rather than what its last message says?为什么评估 Agent 时要看环境的最终状态,而不是看它最后一条回复说了什么?
Common in ChinaCommon overseasBasic#evaluation#graders#outcome🔒 Unlock this chapter to see the answer keys.
When should you report pass@k, and when must you report pass^k? Give an example of each and state the cost.什么场景该用 pass 的 at k,什么场景必须用 pass 的 k 次方?各举一个例子并说明代价。
Common in ChinaCommon overseasIntermediate#evaluation#metrics#pass-at-k🔒 Unlock this chapter to see the answer keys.
A newly written evaluation task has a 0% pass rate over one hundred trials. What is your first reaction?一个新写的评估任务,跑一百次通过率是零。你的第一反应是什么?
Common in ChinaCommon overseasIntermediate#evaluation#debugging#task-quality🔒 Unlock this chapter to see the answer keys.
