Dayward AIWeChat Mini Program
All courses

7 days · about 29 hours

Agent Evals and Observability in 7 Days: Turning "Seems Fine" Into Numbers You Can Reproduce

For anyone who has shipped an agent: build a zero-dependency evaluation framework in seven days. Turn non-determinism into pass^k, mine your incidents for a benchmark set, put the model judge itself through calibration, score the trajectory rather than just the ending, wire up telemetry and a cost panel, and finally block regressions before they merge.

Sign in and link WeChat to share one free article per day across both platforms. Unlock more with rewarded videos in the mini program.

Start day one

By the end

A portfolio-ready evalkit: it reports pass^k, ships a 24-item benchmark set covering both directions, calibrates its judge against human labels, catches detours and loops, traces and prices every run, and turns the pipeline red when quality drops

Hand-build an evaluation framework with no third-party runtime dependencies. Start by turning non-determinism into two probabilities, then add a benchmark set, a model judge with its own calibration, trajectory scoring, telemetry and cost aggregation, and end with a regression gate plus a health check on the suite itself

View account and linked status →7 days · about 29 hours

Week 1

From "Seems Fine" to an Evaluation System That Blocks Regressions

01
D1🔒 Locked

Why You Cannot Ship an Agent on "I Tried It a Few Times"

Start with the three things that make agents harder to evaluate than single-turn systems, settle the five terms this course runs on, and then turn non-determinism into two numbers you can put in a report: the odds of succeeding at least once, and the odds of succeeding every single time.

02
D2🔒 Locked

Benchmark Sets: Turning the Incidents You Already Had Into Reproducible Tasks

The ceiling of an evaluation is set by task quality, not by the metric formula. Today you mine support tickets and failure logs for tasks, settle what makes a task good, add the negative cases almost everyone forgets, and validate every item with a reference solution before it ever judges an agent.

03
D3🔒 Locked

Let a Model Judge, Then Put the Judge Through an Exam

Anything code can decide should never go to a model, but open-ended quality genuinely has to. Today you write rubrics and pairwise comparison, measure the judge's three systematic biases with a position-swap experiment, and decide whether it is fit to use with an agreement coefficient rather than accuracy.

04
D4🔒 Locked

Trajectory Evaluation: It Reached the Destination, but How Many Things Did It Hit

Looking only at the outcome lets a whole class of failures through: the answer was right, but it cost ten times as much, took seven detours, and touched a tool it had no business touching. Today you score the trajectory itself with tool-sequence checks, loop detection and step budgets, then extend single-turn evaluation to multi-turn with a simulated user.

05
D5🔒 Locked

Observability: Putting a Flight Recorder on Every Run

Offline evaluation only covers the tasks you thought of; production is where the real users are. Today you instrument the target against the open generative AI semantic conventions, turn one run into a queryable span tree, aggregate cost and latency, and report the evaluation scores themselves as telemetry.

06
D6🔒 Locked

The Regression Gate: Blocking a Drop Before It Merges

An evaluation only prevents incidents once it runs in the pipeline. Today you freeze a baseline into a snapshot, set a threshold on a score that moves every run without either missing regressions or crying wolf daily, and solve the two problems that actually bite: what a run costs, and who decides whether a red build is real.

07
D7🔒 Locked

Examining the Suite Itself: Saturation, Broken Tasks and Grading Defects

The last day turns the lens on the evaluation system. A suite approaching a perfect score has stopped carrying information, and a single mis-written grader can make a good model look fifty points worse. Today you health-check the suite, establish the discipline of reading transcripts, and assemble seven days of work into a portfolio project.