7 天 Agent 评估与可观测:让「感觉还行」变成可复现的数字
课程题库 · 28 / 28 道题
标签
全部#evaluation20#ci4#observability4#graders3#llm-as-judge3#metrics3#trajectory3#cost2#debugging2#golden-set2#opentelemetry2#process2
还有 33 个标签收起标签
#task-quality2#baseline1#bias1#capability-vs-regression1#culture1#dependency-risk1#grading-defect1#kappa1#licensing1#loop-detection1#multi-turn1#negative-tests1#non-determinism1#outcome1#pairwise1#pass-at-k1#production1#reference-solution1#regression1#reporting1#reproducibility1#sampling1#saturation1#semantic-conventions1#simulated-user1#span-design1#strategy1#suite-health1#task-design1#thresholds1#tooling1#tracing1#transcripts1
D1 为什么 Agent 不能靠「试几次感觉还行」
同一个任务跑五次,三次对两次错。你会怎么向老板汇报这个 Agent 的质量?You run the same task five times and get three passes and two failures. How do you report this agent's quality to your manager?
国内高频海外高频进阶#evaluation#metrics#non-determinism🔒 解锁 D1 后查看答题要点。
为什么评估 Agent 时要看环境的最终状态,而不是看它最后一条回复说了什么?Why should you evaluate an agent against the final state of the environment rather than what its last message says?
国内高频海外高频基础#evaluation#graders#outcome🔒 解锁 D1 后查看答题要点。
什么场景该用 pass 的 at k,什么场景必须用 pass 的 k 次方?各举一个例子并说明代价。When should you report pass@k, and when must you report pass^k? Give an example of each and state the cost.
国内高频海外高频进阶#evaluation#metrics#pass-at-k🔒 解锁 D1 后查看答题要点。
一个新写的评估任务,跑一百次通过率是零。你的第一反应是什么?A newly written evaluation task has a 0% pass rate over one hundred trials. What is your first reaction?
国内高频海外高频进阶#evaluation#debugging#task-quality🔒 解锁 D1 后查看答题要点。
D2 基准集:把线上翻过的车变成可复现的任务
老板让你两天内给一个已上线的 Agent 建评估,你会从哪里找测试任务?Your manager gives you two days to build an evaluation for an agent that is already in production. Where do you get the test tasks?
国内高频海外高频进阶#evaluation#golden-set#task-design🔒 解锁 D2 后查看答题要点。
什么叫单向优化?举例说明只写正向用例会把系统带到什么地方。What is one-sided optimization? Give an example of where a suite containing only positive cases drives a system.
国内高频海外高频进阶#evaluation#golden-set#negative-tests🔒 解锁 D2 后查看答题要点。
一条评估任务应该满足什么条件才算合格?你怎么验证它本身没写错?What makes an evaluation task well-formed, and how do you verify the task itself is not broken?
国内高频海外高频深入#evaluation#task-quality#reference-solution🔒 解锁 D2 后查看答题要点。
能力评估和回归评估的通过率,你分别期望它们是多少?为什么不一样?What pass rates do you expect from a capability suite versus a regression suite, and why are they different?
国内高频海外高频进阶#evaluation#capability-vs-regression#reporting🔒 解锁 D2 后查看答题要点。
D3 让模型当裁判,再把裁判本身考一遍
什么情况下你会拒绝用模型当裁判,改用代码或人工?说出你的判断顺序。When would you refuse to use a model as the judge and reach for code or a human instead? State your decision order.
国内高频海外高频进阶#evaluation#graders#llm-as-judge🔒 解锁 D3 后查看答题要点。
你怎么证明自己的模型裁判没有位置偏好?请描述一个可执行的实验。How would you prove your model judge has no position bias? Describe an experiment someone could actually run.
国内高频海外高频深入#evaluation#llm-as-judge#bias🔒 解锁 D3 后查看答题要点。
为什么评估裁判质量要用一致性系数而不是准确率?举一个准确率会骗人的场景。Why measure judge quality with an agreement coefficient rather than accuracy? Give a scenario where accuracy lies.
国内高频海外高频深入#evaluation#metrics#kappa🔒 解锁 D3 后查看答题要点。
成对比较时两轮结论不一致,你会怎么处理这条数据?为什么不是随机取一个?In a pairwise comparison the two rounds disagree. How do you handle that data point, and why not just pick one at random?
国内高频海外高频进阶#evaluation#llm-as-judge#pairwise🔒 解锁 D3 后查看答题要点。
D4 轨迹评估:它到了终点,但一路上撞了几次
一个 Agent 上线后结果全对,但平均成本翻了十倍。你的评估体系怎么才能发现这件事?An agent ships with all outcomes correct, but its average cost has gone up tenfold. How would your evaluation system catch that?
国内高频海外高频进阶#evaluation#trajectory#cost🔒 解锁 D4 后查看答题要点。
你会把「工具调用顺序必须与参考流程完全一致」写进评分器吗?说出你的理由。Would you assert that an agent's tool-call sequence must exactly match a reference workflow? Give your reasoning.
国内高频海外高频进阶#evaluation#trajectory#graders🔒 解锁 D4 后查看答题要点。
怎么自动判定一个 Agent 陷入了死循环?给出你的规则,以及它的误判风险。How do you automatically detect that an agent is stuck in a loop? Give your rule and its false-positive risks.
国内高频海外高频进阶#evaluation#trajectory#loop-detection🔒 解锁 D4 后查看答题要点。
多轮对话场景怎么做自动化评估?模拟用户会带来什么问题?How do you automate evaluation of multi-turn conversations, and what problems does a simulated user introduce?
国内高频海外高频深入#evaluation#multi-turn#simulated-user🔒 解锁 D4 后查看答题要点。
D5 可观测:给每一次运行装上行车记录仪
为什么给 Agent 埋点要用公开的语义约定,而不是团队自己定一套字段名?Why instrument an agent with a public semantic convention instead of a field-name scheme your team invents?
国内高频海外高频基础#observability#opentelemetry#semantic-conventions🔒 解锁 D5 后查看答题要点。
一个仍在开发中、属性名还会改的标准,你会现在就用吗?怎么控制升级风险?Would you adopt a standard that is still in development and whose attribute names may change? How do you control the upgrade risk?
国内高频海外高频深入#observability#opentelemetry#dependency-risk🔒 解锁 D5 后查看答题要点。
一次 Agent 运行的链路树你会怎么分层?每层各记哪些属性?How would you layer the trace tree for one agent run, and what attributes go on each layer?
国内高频海外高频进阶#observability#tracing#span-design🔒 解锁 D5 后查看答题要点。
线上全量记录太贵,你怎么设计采样策略才不会把真正的故障样本丢掉?Recording every trace in production is too expensive. How do you design sampling so that you do not throw away the failures you actually need?
国内高频海外高频深入#observability#sampling#production🔒 解锁 D5 后查看答题要点。
D6 回归门禁:让退化在合并之前就被拦住
评估分数每次都在抖,你怎么定一个既不漏报也不天天误报的门禁阈值?Evaluation scores fluctuate on every run. How do you set a gate threshold that neither misses regressions nor fires false alarms every day?
国内高频海外高频深入#ci#regression#thresholds🔒 解锁 D6 后查看答题要点。
基线快照里除了通过率,你还会记录什么?少记了会出什么问题?Beyond the pass rate, what else goes into a baseline snapshot? What breaks if you leave something out?
国内高频海外高频进阶#ci#baseline#reproducibility🔒 解锁 D6 后查看答题要点。
每次提交都跑全量评估太贵,你会怎么分层?依据是什么?Running the full evaluation on every commit is too expensive. How would you tier it, and on what basis?
国内高频海外高频进阶#ci#cost#strategy🔒 解锁 D6 后查看答题要点。
评估门禁红了,团队却越来越习惯直接跳过。你会怎么改这条流程?Your evaluation gate keeps firing, and the team has learned to skip it. How would you change the process?
国内高频海外高频深入#ci#process#culture🔒 解锁 D6 后查看答题要点。
D7 体检套件本身:饱和、坏任务与判分缺陷
你们的评估套件通过率已经百分之九十八了,接下来你会做什么?Your evaluation suite is passing at 98 percent. What do you do next?
国内高频海外高频进阶#evaluation#saturation#suite-health🔒 解锁 D7 后查看答题要点。
一个模型在你的套件上突然掉了很多分,你怎么区分是模型退化还是判分写错了?A model suddenly drops a lot of points on your suite. How do you tell a real regression from a broken grader?
国内高频海外高频深入#evaluation#debugging#grading-defect🔒 解锁 D7 后查看答题要点。
为什么说没读过轨迹就不该相信评估分数?你会怎么把读轨迹变成团队习惯?Why should you not trust an evaluation score before someone has read transcripts? And how do you make transcript reading a team habit?
国内高频海外高频进阶#evaluation#transcripts#process🔒 解锁 D7 后查看答题要点。
自建评估框架还是用现成的?说出你的判断依据,以及选型时会看许可的哪些方面。Build your own evaluation framework or adopt an existing one? Give your decision criteria, and say what you check in the license.
国内高频海外高频深入#evaluation#tooling#licensing🔒 解锁 D7 后查看答题要点。