Dayward AIWeChat Mini Program
Agent Evals and Observability in 7 Days: Turning "Seems Fine" Into Numbers You Can Reproduce
Week 1 · D4🔒 LockedAbout 4 hours

Trajectory Evaluation: It Reached the Destination, but How Many Things Did It Hit

Looking only at the outcome lets a whole class of failures through: the answer was right, but it cost ten times as much, took seven detours, and touched a tool it had no business touching. Today you score the trajectory itself with tool-sequence checks, loop detection and step budgets, then extend single-turn evaluation to multi-turn with a simulated user.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Name three failure classes only a trajectory reveals, and give an automatically checkable rule for each
  2. Implement loop detection and a step budget, and explain why the threshold belongs to the task rather than to the whole suite
  3. Run a multi-turn evaluation with a simulated user, and say what bias the simulated user itself introduces

Plain-Language Walkthrough

A dashcam

A car arrives at its destination on time. What does that tell you?

It tells you the car arrived. Whether it ran a red light, circled the same intersection four times, or cut down a one-way street the wrong way — none of that is readable from the fact of arrival.

That is exactly why a dashcam exists. It does not replace the question of whether you got there; arrival is still the first question. What it adds is what happened along the way. After a collision, the insurer asks for the footage, not the itinerary.

The first three days built evaluations that are all itinerary: was the outcome right, how did the judge score it. Today you install the dashcam.

Sign in and link WeChat to read

Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.

Mini program QR code

Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.

Interview questions

  • An agent ships with all outcomes correct, but its average cost has gone up tenfold. How would your evaluation system catch that?一个 Agent 上线后结果全对,但平均成本翻了十倍。你的评估体系怎么才能发现这件事?
    Common in ChinaCommon overseasIntermediate#evaluation#trajectory#cost

    🔒 Unlock this chapter to see the answer keys.

  • Would you assert that an agent's tool-call sequence must exactly match a reference workflow? Give your reasoning.你会把「工具调用顺序必须与参考流程完全一致」写进评分器吗?说出你的理由。
    Common in ChinaCommon overseasIntermediate#evaluation#trajectory#graders

    🔒 Unlock this chapter to see the answer keys.

  • How do you automatically detect that an agent is stuck in a loop? Give your rule and its false-positive risks.怎么自动判定一个 Agent 陷入了死循环?给出你的规则,以及它的误判风险。
    Common in ChinaCommon overseasIntermediate#evaluation#trajectory#loop-detection

    🔒 Unlock this chapter to see the answer keys.

  • How do you automate evaluation of multi-turn conversations, and what problems does a simulated user introduce?多轮对话场景怎么做自动化评估?模拟用户会带来什么问题?
    Common in ChinaCommon overseasDeep dive#evaluation#multi-turn#simulated-user

    🔒 Unlock this chapter to see the answer keys.

Comments