逐日AI微信小程序
← 返回全部课程题库

7 天 Agent 评估与可观测:让「感觉还行」变成可复现的数字

课程题库 · 28 / 28 道题

D1 为什么 Agent 不能靠「试几次感觉还行」

  • 同一个任务跑五次,三次对两次错。你会怎么向老板汇报这个 Agent 的质量?You run the same task five times and get three passes and two failures. How do you report this agent's quality to your manager?
    国内高频海外高频进阶#evaluation#metrics#non-determinism

    🔒 解锁 D1 后查看答题要点。

  • 为什么评估 Agent 时要看环境的最终状态,而不是看它最后一条回复说了什么?Why should you evaluate an agent against the final state of the environment rather than what its last message says?
    国内高频海外高频基础#evaluation#graders#outcome

    🔒 解锁 D1 后查看答题要点。

  • 什么场景该用 pass 的 at k,什么场景必须用 pass 的 k 次方?各举一个例子并说明代价。When should you report pass@k, and when must you report pass^k? Give an example of each and state the cost.
    国内高频海外高频进阶#evaluation#metrics#pass-at-k

    🔒 解锁 D1 后查看答题要点。

  • 一个新写的评估任务,跑一百次通过率是零。你的第一反应是什么?A newly written evaluation task has a 0% pass rate over one hundred trials. What is your first reaction?
    国内高频海外高频进阶#evaluation#debugging#task-quality

    🔒 解锁 D1 后查看答题要点。

D2 基准集:把线上翻过的车变成可复现的任务

  • 老板让你两天内给一个已上线的 Agent 建评估,你会从哪里找测试任务?Your manager gives you two days to build an evaluation for an agent that is already in production. Where do you get the test tasks?
    国内高频海外高频进阶#evaluation#golden-set#task-design

    🔒 解锁 D2 后查看答题要点。

  • 什么叫单向优化?举例说明只写正向用例会把系统带到什么地方。What is one-sided optimization? Give an example of where a suite containing only positive cases drives a system.
    国内高频海外高频进阶#evaluation#golden-set#negative-tests

    🔒 解锁 D2 后查看答题要点。

  • 一条评估任务应该满足什么条件才算合格?你怎么验证它本身没写错?What makes an evaluation task well-formed, and how do you verify the task itself is not broken?
    国内高频海外高频深入#evaluation#task-quality#reference-solution

    🔒 解锁 D2 后查看答题要点。

  • 能力评估和回归评估的通过率,你分别期望它们是多少?为什么不一样?What pass rates do you expect from a capability suite versus a regression suite, and why are they different?
    国内高频海外高频进阶#evaluation#capability-vs-regression#reporting

    🔒 解锁 D2 后查看答题要点。

D3 让模型当裁判,再把裁判本身考一遍

  • 什么情况下你会拒绝用模型当裁判,改用代码或人工?说出你的判断顺序。When would you refuse to use a model as the judge and reach for code or a human instead? State your decision order.
    国内高频海外高频进阶#evaluation#graders#llm-as-judge

    🔒 解锁 D3 后查看答题要点。

  • 你怎么证明自己的模型裁判没有位置偏好?请描述一个可执行的实验。How would you prove your model judge has no position bias? Describe an experiment someone could actually run.
    国内高频海外高频深入#evaluation#llm-as-judge#bias

    🔒 解锁 D3 后查看答题要点。

  • 为什么评估裁判质量要用一致性系数而不是准确率?举一个准确率会骗人的场景。Why measure judge quality with an agreement coefficient rather than accuracy? Give a scenario where accuracy lies.
    国内高频海外高频深入#evaluation#metrics#kappa

    🔒 解锁 D3 后查看答题要点。

  • 成对比较时两轮结论不一致,你会怎么处理这条数据?为什么不是随机取一个?In a pairwise comparison the two rounds disagree. How do you handle that data point, and why not just pick one at random?
    国内高频海外高频进阶#evaluation#llm-as-judge#pairwise

    🔒 解锁 D3 后查看答题要点。

D4 轨迹评估:它到了终点,但一路上撞了几次

  • 一个 Agent 上线后结果全对,但平均成本翻了十倍。你的评估体系怎么才能发现这件事?An agent ships with all outcomes correct, but its average cost has gone up tenfold. How would your evaluation system catch that?
    国内高频海外高频进阶#evaluation#trajectory#cost

    🔒 解锁 D4 后查看答题要点。

  • 你会把「工具调用顺序必须与参考流程完全一致」写进评分器吗?说出你的理由。Would you assert that an agent's tool-call sequence must exactly match a reference workflow? Give your reasoning.
    国内高频海外高频进阶#evaluation#trajectory#graders

    🔒 解锁 D4 后查看答题要点。

  • 怎么自动判定一个 Agent 陷入了死循环?给出你的规则,以及它的误判风险。How do you automatically detect that an agent is stuck in a loop? Give your rule and its false-positive risks.
    国内高频海外高频进阶#evaluation#trajectory#loop-detection

    🔒 解锁 D4 后查看答题要点。

  • 多轮对话场景怎么做自动化评估?模拟用户会带来什么问题?How do you automate evaluation of multi-turn conversations, and what problems does a simulated user introduce?
    国内高频海外高频深入#evaluation#multi-turn#simulated-user

    🔒 解锁 D4 后查看答题要点。

D5 可观测:给每一次运行装上行车记录仪

  • 为什么给 Agent 埋点要用公开的语义约定,而不是团队自己定一套字段名?Why instrument an agent with a public semantic convention instead of a field-name scheme your team invents?
    国内高频海外高频基础#observability#opentelemetry#semantic-conventions

    🔒 解锁 D5 后查看答题要点。

  • 一个仍在开发中、属性名还会改的标准,你会现在就用吗?怎么控制升级风险?Would you adopt a standard that is still in development and whose attribute names may change? How do you control the upgrade risk?
    国内高频海外高频深入#observability#opentelemetry#dependency-risk

    🔒 解锁 D5 后查看答题要点。

  • 一次 Agent 运行的链路树你会怎么分层?每层各记哪些属性?How would you layer the trace tree for one agent run, and what attributes go on each layer?
    国内高频海外高频进阶#observability#tracing#span-design

    🔒 解锁 D5 后查看答题要点。

  • 线上全量记录太贵,你怎么设计采样策略才不会把真正的故障样本丢掉?Recording every trace in production is too expensive. How do you design sampling so that you do not throw away the failures you actually need?
    国内高频海外高频深入#observability#sampling#production

    🔒 解锁 D5 后查看答题要点。

D6 回归门禁:让退化在合并之前就被拦住

  • 评估分数每次都在抖,你怎么定一个既不漏报也不天天误报的门禁阈值?Evaluation scores fluctuate on every run. How do you set a gate threshold that neither misses regressions nor fires false alarms every day?
    国内高频海外高频深入#ci#regression#thresholds

    🔒 解锁 D6 后查看答题要点。

  • 基线快照里除了通过率,你还会记录什么?少记了会出什么问题?Beyond the pass rate, what else goes into a baseline snapshot? What breaks if you leave something out?
    国内高频海外高频进阶#ci#baseline#reproducibility

    🔒 解锁 D6 后查看答题要点。

  • 每次提交都跑全量评估太贵,你会怎么分层?依据是什么?Running the full evaluation on every commit is too expensive. How would you tier it, and on what basis?
    国内高频海外高频进阶#ci#cost#strategy

    🔒 解锁 D6 后查看答题要点。

  • 评估门禁红了,团队却越来越习惯直接跳过。你会怎么改这条流程?Your evaluation gate keeps firing, and the team has learned to skip it. How would you change the process?
    国内高频海外高频深入#ci#process#culture

    🔒 解锁 D6 后查看答题要点。

D7 体检套件本身:饱和、坏任务与判分缺陷

  • 你们的评估套件通过率已经百分之九十八了,接下来你会做什么?Your evaluation suite is passing at 98 percent. What do you do next?
    国内高频海外高频进阶#evaluation#saturation#suite-health

    🔒 解锁 D7 后查看答题要点。

  • 一个模型在你的套件上突然掉了很多分,你怎么区分是模型退化还是判分写错了?A model suddenly drops a lot of points on your suite. How do you tell a real regression from a broken grader?
    国内高频海外高频深入#evaluation#debugging#grading-defect

    🔒 解锁 D7 后查看答题要点。

  • 为什么说没读过轨迹就不该相信评估分数?你会怎么把读轨迹变成团队习惯?Why should you not trust an evaluation score before someone has read transcripts? And how do you make transcript reading a team habit?
    国内高频海外高频进阶#evaluation#transcripts#process

    🔒 解锁 D7 后查看答题要点。

  • 自建评估框架还是用现成的?说出你的判断依据,以及选型时会看许可的哪些方面。Build your own evaluation framework or adopt an existing one? Give your decision criteria, and say what you check in the license.
    国内高频海外高频深入#evaluation#tooling#licensing

    🔒 解锁 D7 后查看答题要点。