Trajectory Evaluation: It Reached the Destination, but How Many Things Did It Hit
Looking only at the outcome lets a whole class of failures through: the answer was right, but it cost ten times as much, took seven detours, and touched a tool it had no business touching. Today you score the trajectory itself with tool-sequence checks, loop detection and step budgets, then extend single-turn evaluation to multi-turn with a simulated user.
Today's Goals
- Name three failure classes only a trajectory reveals, and give an automatically checkable rule for each
- Implement loop detection and a step budget, and explain why the threshold belongs to the task rather than to the whole suite
- Run a multi-turn evaluation with a simulated user, and say what bias the simulated user itself introduces
Plain-Language Walkthrough
A dashcam
A car arrives at its destination on time. What does that tell you?
It tells you the car arrived. Whether it ran a red light, circled the same intersection four times, or cut down a one-way street the wrong way — none of that is readable from the fact of arrival.
That is exactly why a dashcam exists. It does not replace the question of whether you got there; arrival is still the first question. What it adds is what happened along the way. After a collision, the insurer asks for the footage, not the itinerary.
The first three days built evaluations that are all itinerary: was the outcome right, how did the judge score it. Today you install the dashcam.
Interview questions
An agent ships with all outcomes correct, but its average cost has gone up tenfold. How would your evaluation system catch that?一个 Agent 上线后结果全对,但平均成本翻了十倍。你的评估体系怎么才能发现这件事?
Common in ChinaCommon overseasIntermediate#evaluation#trajectory#cost🔒 Unlock this chapter to see the answer keys.
Would you assert that an agent's tool-call sequence must exactly match a reference workflow? Give your reasoning.你会把「工具调用顺序必须与参考流程完全一致」写进评分器吗?说出你的理由。
Common in ChinaCommon overseasIntermediate#evaluation#trajectory#graders🔒 Unlock this chapter to see the answer keys.
How do you automatically detect that an agent is stuck in a loop? Give your rule and its false-positive risks.怎么自动判定一个 Agent 陷入了死循环?给出你的规则,以及它的误判风险。
Common in ChinaCommon overseasIntermediate#evaluation#trajectory#loop-detection🔒 Unlock this chapter to see the answer keys.
How do you automate evaluation of multi-turn conversations, and what problems does a simulated user introduce?多轮对话场景怎么做自动化评估?模拟用户会带来什么问题?
Common in ChinaCommon overseasDeep dive#evaluation#multi-turn#simulated-user🔒 Unlock this chapter to see the answer keys.
