Benchmark Sets: Turning the Incidents You Already Had Into Reproducible Tasks
The ceiling of an evaluation is set by task quality, not by the metric formula. Today you mine support tickets and failure logs for tasks, settle what makes a task good, add the negative cases almost everyone forgets, and validate every item with a reference solution before it ever judges an agent.
The ceiling of an evaluation is set by task quality, not by the metric formula. Today you mine support tickets and failure logs for tasks, settle what makes a task good, add the negative cases almost everyone forgets, and validate every item with a reference solution before it ever judges an agent.
Interview questions
Your manager gives you two days to build an evaluation for an agent that is already in production. Where do you get the test tasks?老板让你两天内给一个已上线的 Agent 建评估,你会从哪里找测试任务?
Common in ChinaCommon overseasIntermediate#evaluation#golden-set#task-design🔒 Unlock this chapter to see the answer keys.
What is one-sided optimization? Give an example of where a suite containing only positive cases drives a system.什么叫单向优化?举例说明只写正向用例会把系统带到什么地方。
Common in ChinaCommon overseasIntermediate#evaluation#golden-set#negative-tests🔒 Unlock this chapter to see the answer keys.
What makes an evaluation task well-formed, and how do you verify the task itself is not broken?一条评估任务应该满足什么条件才算合格?你怎么验证它本身没写错?
Common in ChinaCommon overseasDeep dive#evaluation#task-quality#reference-solution🔒 Unlock this chapter to see the answer keys.
What pass rates do you expect from a capability suite versus a regression suite, and why are they different?能力评估和回归评估的通过率,你分别期望它们是多少?为什么不一样?
Common in ChinaCommon overseasIntermediate#evaluation#capability-vs-regression#reporting🔒 Unlock this chapter to see the answer keys.
