Dayward AIWeChat Mini Program
Agent Evals and Observability in 7 Days: Turning "Seems Fine" Into Numbers You Can Reproduce
Week 1 · D5🔒 LockedAbout 4 hours

Observability: Putting a Flight Recorder on Every Run

Offline evaluation only covers the tasks you thought of; production is where the real users are. Today you instrument the target against the open generative AI semantic conventions, turn one run into a queryable span tree, aggregate cost and latency, and report the evaluation scores themselves as telemetry.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Instrument a model call and a tool call against the open semantic conventions, and explain why the convention version has to be pinned in your own code
  2. Aggregate cost, latency and failure rate per task from span data, and say which field time-to-first-chunk belongs in
  3. Report evaluation scores as telemetry attributes, and explain how that stitches offline evaluation together with production monitoring

Plain-Language Walkthrough

Timestamps on every leg of a flight

You fly from Shanghai to Chengdu with a connection in Xi'an. The itinerary does not say "Shanghai to Chengdu, eight hours." It says it as timestamped legs: check-in, first flight out and in, the wait at the connection, second flight out and in, bags on the belt.

Those legs have three properties, and they map onto everything in today's material.

First, each leg has its own start and end. When the trip runs eight hours long, you can point at the leg that ate the time — a delay on the first flight, or three hours in the connection queue.

Second, the legs nest rather than sit side by side. Boarding the second flight belongs inside that flight's leg. Nesting is what lets you say how long "the connection" took as a whole, instead of stitching a flat pile of events back together by timestamp.

Third, each leg carries attributes. Flight number, aircraft type, gate. Attributes exist for aggregation: the average taxi time for that aircraft type at that airport is answerable only because of them.

After four days you already have a transcript — the complete record of one trial. Today's job is to turn a run into a timestamped, nested, attributed itinerary, written with field names other people already recognize.

Sign in and link WeChat to read

Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.

Mini program QR code

Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.

Interview questions

  • Why instrument an agent with a public semantic convention instead of a field-name scheme your team invents?为什么给 Agent 埋点要用公开的语义约定,而不是团队自己定一套字段名?
    Common in ChinaCommon overseasBasic#observability#opentelemetry#semantic-conventions

    🔒 Unlock this chapter to see the answer keys.

  • Would you adopt a standard that is still in development and whose attribute names may change? How do you control the upgrade risk?一个仍在开发中、属性名还会改的标准,你会现在就用吗?怎么控制升级风险?
    Common in ChinaCommon overseasDeep dive#observability#opentelemetry#dependency-risk

    🔒 Unlock this chapter to see the answer keys.

  • How would you layer the trace tree for one agent run, and what attributes go on each layer?一次 Agent 运行的链路树你会怎么分层?每层各记哪些属性?
    Common in ChinaCommon overseasIntermediate#observability#tracing#span-design

    🔒 Unlock this chapter to see the answer keys.

  • Recording every trace in production is too expensive. How do you design sampling so that you do not throw away the failures you actually need?线上全量记录太贵,你怎么设计采样策略才不会把真正的故障样本丢掉?
    Common in ChinaCommon overseasDeep dive#observability#sampling#production

    🔒 Unlock this chapter to see the answer keys.

Comments