Trajectory Evaluation: It Reached the Destination, but How Many Things Did It Hit

Interview questions

• An agent ships with all outcomes correct, but its average cost has gone up tenfold. How would your evaluation system catch that?一个 Agent 上线后结果全对，但平均成本翻了十倍。你的评估体系怎么才能发现这件事？

  Common in ChinaCommon overseasIntermediate#evaluation#trajectory#cost

  How to reason about it · think before answering

  1. This tests whether you understand that outcome-based evaluation has a structural blind spot. Answering 'add a cost alert' earns half credit - that is remediation after the fact, while the question asks why the evaluation itself missed it.
  2. State the mechanism: an outcome grader reads the final state of the environment, and cost is not part of that state. The refunds table holds an order id and an amount, not the tokens spent or the number of tool calls. No matter how strict your outcome assertions are, they cannot in principle detect a tenfold cost increase. That is a coverage gap, not an oversight.
  3. The fix is to assert on the trajectory itself: total tool calls per trajectory below N, total spend below M. These sit in a layer parallel to the outcome grader, and either breach fails the trial. It must be a failure rather than a warning - warnings in CI are equivalent to nothing.
  4. Where the thresholds come from: the current baseline. Measure the present distribution over a batch of trials and set the ceiling slightly above today's 95th percentile rather than picking a round number. That tolerates normal jitter while going red as soon as the mean shifts up.
  5. Add why this class of regression slips into production so easily: cost regressions change nothing a user can see. A new prompt, one more reflection round, longer tool descriptions - the pass rate is unchanged, steps go from three to eight, and the bill surfaces at month end. Without budget assertions the pipeline stays green the whole way.
  6. Expected follow-up: steps or spend as the gate? Both, because they diverge. Swapping in a pricier but smarter model lowers steps and raises unit price. Watching only steps misses the price increase; watching only spend misses the extra wandering caused by weaker reasoning.

  分析过程 · 先想清楚再作答

  1. 这题考的是「知不知道结果态评估有结构性盲区」。回答「加一个成本监控告警」只能拿一半分——那是发现之后的补救，题目问的是评估体系本身为什么漏掉了它。
  2. 先把机制说清楚：结果态评分器的输入是环境的最终状态，而成本不在最终状态里。退款记录表里只有订单号和金额，没有「这次花了多少 token、调了几次工具」。所以无论你把结果态断言写得多严，它在原理上都判不出成本翻十倍这件事。**这是覆盖不到，不是写漏了。**
  3. 正确的做法是给轨迹本身写断言：一条轨迹的工具调用总数不超过 N、总花费不超过 M。这两条挂在与结果态平行的一层，任一超标就判这次试次失败。注意它必须是**失败**而不是警告——警告在 CI 里等于没有。
  4. 阈值从哪来：从当前基线来。先跑一批试次统计出现在的分布，取一个略高于当前 p95 的数作为上限，而不是拍一个整数。这样它既能容忍正常抖动，又能在均值整体上移时立刻报红。
  5. 还要补一句为什么这类退化特别容易溜进生产：**成本回归不改变任何用户可见的行为。** 换个提示词、多加一轮反思、把工具描述写长一点，成功率一点没掉，步数从 3 涨到 8，账单要到月底才有人看。没有预算断言的话，评估流水线全程报绿。
  6. 可预期的追问是「那步数和花费该选哪个当闸门」。答案是两个都要，因为它们会分叉：模型换成一个更贵但更聪明的，步数会降而单价会升。只看步数会漏掉换模型带来的涨价，只看花费会漏掉逻辑变笨带来的绕路。

  Key points

  • Cost is absent from the outcome state, so an outcome grader cannot detect this class of regression at all.
  • Add a parallel layer of trajectory assertions: a step ceiling and a spend ceiling, failing the trial rather than warning.
  • Derive thresholds from the current baseline distribution, slightly above p95, not from a round number.
  • Cost regressions change nothing user-visible, so without budget assertions the pipeline stays green.
  • Track both steps and spend: a pricier model moves them in opposite directions and either alone leaves a hole.

  答题要点

  • 结果态里根本不含成本，所以结果态评分器在原理上覆盖不到这类退化。
  • 给轨迹写平行的一层断言：步数上限与花费上限，超标判失败而不是告警。
  • 阈值从当前基线的分布取，略高于 p95，而不是拍一个整数。
  • 成本回归不改变任何用户可见行为，所以没有预算断言时流水线会全程报绿。
  • 步数与花费都要盯：换更贵的模型会让两者反向变化，只看一个都会漏。
• Would you assert that an agent's tool-call sequence must exactly match a reference workflow? Give your reasoning.你会把「工具调用顺序必须与参考流程完全一致」写进评分器吗？说出你的理由。

  Common in ChinaCommon overseasIntermediate#evaluation#trajectory#graders

  How to reason about it · think before answering

  1. This is a position question, but the credit is not in the position. Either answer can score; what matters is naming the concrete mechanism of false failures and proposing a workable middle ground.
  2. Why a total-order assertion is wrong: part of an agent's value is finding solutions the designer did not anticipate - one extra order lookup to confirm the amount (more careful), two independent lookups issued concurrently (faster), reading a cached policy conclusion and skipping a call (cheaper). A strict sequence assertion fails all three, and all three are improvements. A grader that punishes improvement is the worst kind of defect: it pushes the team toward a dumber but more obedient implementation.
  3. There is a subtler cost too. Total-order assertions go red en masse on every model upgrade for reasons unrelated to quality. The team then either spends days updating reference workflows or disables the whole class of assertion, losing the real problems it would have caught.
  4. You still cannot assert nothing, or a refund issued without a policy check goes unnoticed. The middle ground is to assert only necessary precedence, and as a partial order: before the first issue_refund, check_policy must have appeared at least once. Intervening calls are fine, three policy checks are fine, and if no refund happened the rule is simply skipped.
  5. A simple test for whether a precedence rule belongs in the policy: if it were violated, would a real loss follow? Refunding without a policy check loses money, so it belongs. Whether the order lookup precedes the policy check has no consequence, so it does not. More than about five precedence rules in one policy almost certainly means overreach.
  6. Expected follow-up: if the order was wrong but the outcome was right, does the trial pass? Report them separately - outcome passed, sequence failed, each recorded on its own. Collapsing them into one score destroys exactly the information you need, which is that this run got lucky.

  分析过程 · 先想清楚再作答

  1. 这是一道立场题，但分不在立场上——答「会」或者「不会」都能拿分，关键是能不能说出误杀的**具体机制**，以及给出一个能落地的中间方案。
  2. 先说为什么不该写死全序。Agent 的价值有一部分正来自它会找到设计者没想到的解法：为了确认金额多查一次订单（更谨慎）、把两次独立查询并发发出（更快）、从缓存里直接读到结论省掉一次调用（更省）。这三种在全序断言下**全部判失败**，而它们全是改进。你的评分器在惩罚改进，这是最坏的一种评估缺陷——它会把团队推向一个更笨但更听话的实现。
  3. 还有一个更隐蔽的代价：全序断言会在模型升级时大面积变红，而红的原因与质量无关。于是团队要么花大量时间逐条更新参考流程，要么干脆把这类断言整体关掉，连同它本来能抓到的真问题一起。
  4. 但也不能一条都不判，否则「没查政策就退款」这种真问题没人管。中间方案是**只判必要的前置关系，而且判偏序**：断言「第一次 issue_refund 之前，check_policy 至少出现过一次」。中间夹了别的调用不算违规，查了三次也不算，压根没退款时这条规则直接跳过。
  5. 判断一条前置关系该不该写进去，有个简单标准：**如果它被违反，会不会导致一次真实的损失？** 不查政策就退款会退错钱，该写；先查订单再查政策还是反过来，没有任何后果，不该写。一份策略里超过五条前置关系，基本可以确定写多了。
  6. 可预期的追问是「那顺序错了但结果对了，到底算不算通过」。答案是分开报：结果态通过、序列不通过，两个分数各自记录。合成一个总分会丢掉信息——你需要知道的恰恰是「这次是蒙对的」。

  Key points

  • Do not assert a total order: an extra lookup, concurrent calls, or a cache shortcut all get failed, and all are improvements.
  • Total-order assertions also go red wholesale on model upgrades and end up disabled, taking the real findings with them.
  • The middle ground is necessary precedence as a partial order: a policy check somewhere before the first refund.
  • The test is whether a violation causes real loss; more than about five precedence rules means overreach.
  • Report outcome and sequence as separate scores - merging them hides the fact that a run got lucky.

  答题要点

  • 不写死全序：多查一次、并发查询、走缓存捷径都会被误杀，而它们全是改进。
  • 全序断言还会在模型升级时大面积变红，最终被整体关掉，真问题一起丢掉。
  • 中间方案是只判必要前置且判偏序：第一次退款之前查过政策即可。
  • 取舍标准是「违反了会不会造成真实损失」，超过五条前置关系基本是写多了。
  • 结果态与序列两个分数分开报，合成总分会丢掉「这次是蒙对的」这条关键信息。
• How do you automatically detect that an agent is stuck in a loop? Give your rule and its false-positive risks.怎么自动判定一个 Agent 陷入了死循环？给出你的规则，以及它的误判风险。

  Common in ChinaCommon overseasIntermediate#evaluation#trajectory#loop-detection

  How to reason about it · think before answering

  1. The discrimination here is all in the details. 'More than three calls to the same tool' is the most common answer and it is wrong - it fails a large class of perfectly normal tasks.
  2. The correct criterion is the same tool with identical arguments occurring more than N times. The arguments clause is essential: looking up ten different orders is ten lookup calls with ten different argument sets, which is normal batch work; four lookups of the same order id is a loop.
  3. One implementation detail you must hit: sort the argument keys before serializing them into a signature. Serializing the object directly means the same call written with two key orders produces two different signatures, and the loop goes undetected - a silent failure that raises no error.
  4. N must be set per task, and that is the real point of the question. A task that resolves in one lookup is looping if it repeats three times; polling an asynchronous job legitimately requires a dozen checks. With both kinds in one suite, a single global threshold forces a choice between missing half the loops and failing half the normal tasks.
  5. Three main false-positive risks: legitimate retries after a transient failure, which argues for including success/failure in the signature; idempotent polling, handled by per-task thresholds; and arguments containing timestamps or random ids, which make every signature unique and hide loops entirely - those volatile fields must be stripped when computing the signature.
  6. Expected follow-up: what other shapes of looping exist? Semantic loops, where tools and arguments differ but the agent oscillates between two states. Those are caught by the step budget instead - you cannot prove it is a loop, but you can prove it blew the budget, and for evaluation purposes the conclusion is the same: this trial does not pass.

  分析过程 · 先想清楚再作答

  1. 这题的区分度全在细节上。答「同一个工具调用超过三次就是循环」是最常见的答案，也是错的——它会把一大批正常任务误判成死循环。
  2. 正确的判据是**同一个工具加上完全相同的参数**出现超过 N 次。参数这半句不能省：依次查询十个不同的订单，是十次 lookup 调用但参数各不相同，它是正常的批量操作；连续四次查同一个订单号，才是循环。
  3. 实现上有个必须踩到的细节：参数要**按键排序后**再序列化成签名。直接对参数对象做 JSON 序列化的话，同一次调用写成两种键顺序会算出两个不同签名，于是循环恰好检测不出来——而这是一个不会报错的静默失效。
  4. N 的取值必须**按任务定**，这是这题真正的考点。查一次就有结果的任务，重复三次一定是循环；而轮询一个异步任务的状态，本来就要查十几次才等到完成。同一套评估里两种任务共存，一个全局阈值只能在「漏掉一半死循环」和「误伤一半正常任务」之间挑一个。
  5. 误判风险主要有三类：① 合法的重试——网络失败后重试同一个调用是正确行为，所以理想情况下签名里应该带上返回是否成功；② 幂等的轮询，靠按任务调阈值解决；③ 参数里带了时间戳或随机 id，导致每次签名都不同，循环被完全漏掉，这一类要在算签名时显式剔除易变字段。
  6. 可预期的追问是「除了重复调用，还有什么循环形态」。答案是语义层面的循环：工具和参数都不同，但 Agent 在 A 和 B 两个状态之间来回横跳。这种要靠步数预算兜底——判不出它是循环，但能判出它超了预算，而对评估来说结论是一样的：这次试次不合格。

  Key points

  • The rule is the same tool plus identical arguments exceeding N, not the same tool exceeding N.
  • Sort argument keys when computing the signature, or differing key order silently hides the loop.
  • Set the threshold per task: one-shot lookups and polling workflows cannot share a number.
  • Three false-positive sources: legitimate retries, idempotent polling, and volatile fields such as timestamps or random ids.
  • Semantic loops that oscillate between states are not detectable this way; the step budget catches them instead.

  答题要点

  • 判据是「同一工具 + 完全相同参数」超过 N 次，不是「同一个工具」超过 N 次。
  • 算签名时参数必须按键排序，否则键顺序不同会让循环静默漏检。
  • 阈值按任务定：一次查询就有结果的任务与需要轮询的任务不能共用一个数。
  • 三类误判：合法重试、幂等轮询、参数里带时间戳或随机 id 导致签名永不重复。
  • 语义循环（在两个状态间横跳）检测不出来，靠步数预算兜底。
• How do you automate evaluation of multi-turn conversations, and what problems does a simulated user introduce?多轮对话场景怎么做自动化评估？模拟用户会带来什么问题？

  Common in ChinaCommon overseasDeep dive#evaluation#multi-turn#simulated-user

  How to reason about it · think before answering

  1. The question has two halves and the second is the discriminator. Almost everyone can answer 'write a simulated user'; the score depends on naming the systematic biases it introduces.
  2. Why it is necessary: real users open with incomplete requests and then push back, and the pushback is where agents break - context grows, constraints get diluted, tools get called repeatedly. Single-turn evaluation structurally cannot reach that path, because nobody says 'check again' in their opening sentence.
  3. Two implementations exist. A rule-based user keys off phrases in the agent's last reply, giving perfect reproducibility at zero cost. A model-based user has another model play the customer, giving varied phrasing that is closer to reality.
  4. The rule-based bias is uniformity: it pushes back with the same sentence every time, so the evaluation surfaces exactly one failure mode. Real users express the same intent a hundred ways, and some of those phrasings take entirely different paths.
  5. The model-based bias is subtler - leakage. The model playing the user holds the full task setup and readily gives the answer away in its follow-up ('check clause three of the policy table'), so the agent under test is taking an open-book exam. Both kinds share a third bias: they are too patient. They never hang up, change their mind, or get angry, and abandonment in real conversations is itself an important signal.
  6. Crucially all three biases point the same way: they inflate the score. So multi-turn evaluation is a tool for finding failures, not for estimating success rates. Do not publish the simulated-user pass rate, but do go fix the loop it uncovered. Reporting a rate requires calibration against replayed real conversations or human sampling.
  7. Expected follow-up: how do you make multi-turn evaluation reproducible? Three things - seed the simulated user's randomness, hard-cap the number of turns as a termination condition, and rebuild the environment between trials. Note that within a single conversation the environment is shared across turns; the isolation boundary is the trial, not the turn.

  分析过程 · 先想清楚再作答

  1. 这题分两半，后半半才是考点。前半半答「写一个模拟用户」几乎人人会答，能不能说出它引入的系统性偏差才分高下。
  2. 先讲为什么非做不可：真实用户第一句话往往不完整，会追问，而追问才是最容易翻车的地方——上下文变长、约束被冲淡、工具被反复调用。**单轮评估在结构上碰不到这条路径**，因为没有人会在第一句话里说「你再查一次」。
  3. 实现上有两种模拟用户。规则型看 Agent 上一句回复里的关键词决定下一句说什么，优点是完全可复现、零成本；模型型让另一个模型扮演用户，优点是表达多样、更接近真实。
  4. 规则型的偏差是**表达过于一致**：它每次都用同一句话追问，于是评估只能发现一种失败模式。真实用户会用一百种说法表达同一个意思，其中某些说法会触发完全不同的路径。
  5. 模型型的偏差更隐蔽，是**泄题**：扮演用户的模型拿到的是任务的完整设定，它很容易在追问里把答案说出来（「你去查一下政策表第三条」），于是被评的 Agent 在开卷考试。另外两种模拟用户共有一条偏差：它们**太有耐心**，不会挂电话、不会改主意、不会骂人，而真实对话里的放弃行为本身是一个重要信号。
  6. 关键是这三条偏差**方向一致，都让分数偏高**。所以结论是：多轮评估是发现失败的工具，不是估计成功率的工具。模拟用户跑出的成功率不要直接对外报，但它抓出来的那条死循环可以直接拿去修。要报成功率就得用真实对话回放或人工抽查来校准。
  7. 可预期的追问是「怎么让多轮评估可复现」。答案是三件事：模拟用户的随机源要固定种子、终止条件要写死最大轮数、每次试次之间环境必须重建；而**同一次会话内部的轮次之间环境是共享的**——隔离的边界是试次，不是轮次。

  Key points

  • It is necessary because the follow-up path is structurally unreachable in single-turn evaluation and is where agents break.
  • A rule-based simulated user is reproducible but too uniform, surfacing only one failure mode.
  • A model-based simulated user leaks the answer, since it holds the full task setup.
  • Both are too patient: they never abandon or change their mind, erasing a real signal.
  • All biases inflate the score, so use multi-turn evaluation to find failures, not to report success rates.

  答题要点

  • 必须做的理由：追问路径在单轮评估里结构性地碰不到，而它正是最容易翻车的地方。
  • 规则型模拟用户可复现但表达过于一致，只能发现一种失败模式。
  • 模型型模拟用户会泄题：它拿着完整任务设定，容易在追问里把答案说出来。
  • 两者共有的偏差是太有耐心：不会放弃、不会改主意，抹掉了真实的放弃信号。
  • 三条偏差方向一致地抬高分数，所以多轮评估用于发现失败，不用于报成功率。

Comments