Evaluation and Observability: a Golden Set, LLM-as-Judge, Tracing, a Failure-Rate/Cost Dashboard; Pi vs. LangGraph Summary; Week Three Retrospective
Build an agent evaluation system: score with a golden set and LLM-as-judge, wire in tracing and a failure-rate/cost dashboard, and summarize when Pi and LangGraph each fit, wrapping up week three.
Today's Goals
- Build a small golden set covering typical inputs and expected outputs
- Implement an LLM-as-judge scoring script that scores output and gives a reason
- Wire in tracing and produce an observability dashboard that includes failure rate and cost
Yesterday's closing line is today's starting point: this system triages, splits tasks and runs them in parallel, reviews itself, and reaches out proactively with restraint, and all you have about how well it does any of it is "I tried a few and it seemed fine." Today that sentence becomes numbers.
Plain-Language Walkthrough
Quality control cannot rest on one veteran's hands
When a production line first opens, quality control is often one veteran: they pick up a piece, squeeze it, and say this batch will do. At low volume that works, because they genuinely know; the problem is that the method keeps no record, cannot be reproduced, and changes standard when the person changes — so when a customer complains that last month's goods were better, nobody can say whether that is true.
A mature line grows three things, and those three are what we fit to an agent today: reference samples (a small fixed batch, where only numbers from the same batch compare) are a golden set; an automatic gauge (fast, cheap, and drifting, so it needs calibrating against the samples) is LLM-as-judge; and a routing card (which stations a product passed, who worked on it, how long each took) is tracing.
Why do ordinary backends get by on unit tests while an agent does not? Because the same input does not guarantee the same output. A traditional test asserts equality, and an agent's output has no single correct answer, only good enough. That difference cascades into three consequences.
One, running successfully says nothing about quality. D17's review loop completing means only that the flow threw nothing; it says nothing about whether the output states the refund conclusion. Two, a change's impact is diffuse. A one-word prompt edit may affect one category, and five hand-picked samples that all miss it tell you there was no impact, and you ship. Three, multi-agent is harder again: one request passes routing, splitting, parallel execution, review, and assembly, and any station going crooked presents as "that last paragraph is a bit off," so without measuring them apart you cannot tell which piece to fix.
So the first conclusion is: evaluation is not testing, evaluation establishes a comparable baseline for a stochastic system. Its product is not pass or fail, it is a number comparable with last time. And to compare, the samples must be fixed — so how should they be chosen, and how many?
Interview questions
How do you evaluate an agent's quality, and how does it differ from testing a conventional backend service?怎么评估一个 Agent 的效果?和传统后端服务的测试有什么不同?
Common in ChinaCommon overseasIntermediate#evaluation#testing#agent-quality🔒 Unlock this chapter to see the answer keys.
What makes LLM-as-judge unreliable, and what do you do about it?用大模型给大模型的输出打分(LLM-as-judge),有哪些不可靠的地方?怎么办?
Common in ChinaCommon overseasDeep dive#evaluation#llm-as-judge#reliability🔒 Unlock this chapter to see the answer keys.
What does observability look like for a multi-agent system, and how does it differ from a single agent?多 Agent 系统的可观测性要看哪些东西?和单 Agent 有什么不一样?
Common in ChinaCommon overseasIntermediate#observability#tracing#distributed-systems🔒 Unlock this chapter to see the answer keys.
When should you reach for an orchestration framework like LangGraph, and when should you not?什么时候该用 LangGraph 这类编排框架,什么时候不该用?
Common in ChinaCommon overseasIntermediate#architecture#framework-selection#langgraph🔒 Unlock this chapter to see the answer keys.
System design: a multi-agent support platform is live, the team edits prompts several times a week, nobody can say whether quality is improving, and cost is only known as a month-end total. Design its evaluation and observability system.系统设计:一个多 Agent 客服平台已经上线,团队每周改几次提示词,但没人说得清质量是变好还是变差,成本也只有一个月底的总数。请为它设计一套评估与可观测体系。
Common in ChinaCommon overseasDeep dive#system-design#evaluation#observability#cost🔒 Unlock this chapter to see the answer keys.
