Observability: Putting a Flight Recorder on Every Run

Interview questions

• Why instrument an agent with a public semantic convention instead of a field-name scheme your team invents?为什么给 Agent 埋点要用公开的语义约定，而不是团队自己定一套字段名？

  Common in ChinaCommon overseasBasic#observability#opentelemetry#semantic-conventions

  How to reason about it · think before answering

  1. This tests whether you have ever actually migrated an observability backend. 'For standardization' is an empty answer; give three reasons that map to concrete cost.
  2. First, migration cost. Backends get swapped often - self-hosted to commercial, one vendor to another, or two running side by side for comparison. With a public convention you change one export endpoint; with private field names you change every instrumentation site plus every dashboard query. At scale that difference is an order of magnitude.
  3. Second, ready-made views. Backends ship built-in panels keyed on these fields: spend by model, error rate by tool, traces by conversation. Match the names and the charts exist for free; miss them and you rebuild each one, and new teammates cannot read your private schema.
  4. Third, and most overlooked: the convention is a checklist someone already thought through for you. Left to yourself you record model, latency and token counts. The convention also has cache-read tokens, cache-creation tokens, reasoning tokens, finish reasons, tool call ids. Omitting them raises no error, but when someone asks how much caching saved last month, the history simply is not there - telemetry gaps cannot be backfilled.
  5. State the boundary too: not everything belongs in the public namespace. Your own dimensions - task id, tenant, experiment arm - go under your own prefix. Do not smuggle private fields under gen_ai, or you will not be able to tell yours from theirs when the convention moves.
  6. Expected follow-up: what if the field you need is not in the convention? Check carefully that it really is absent, put it in your own namespace, and watch upstream discussions - renaming once after it lands is cheaper than inventing your own forever.

  分析过程 · 先想清楚再作答

  1. 这题考的是「有没有真的换过可观测后端」。只答「为了标准化」是一句空话，要给出三条能落到具体成本的理由。
  2. 第一条是换后端的成本。可观测后端是换起来很频繁的东西——自建换商业的、商业的换一家、或者双跑做对比。用公开约定时，换后端改的是导出地址一处；用自定字段时，改的是每一处埋点，外加每一张面板的查询语句。这个差别在系统大起来之后是数量级的。
  3. 第二条是现成视图。各家后端都按这套约定做了开箱即用的面板：按模型看花费、按工具看错误率、按会话串链路。字段名对上了这些图不用配就有；对不上就得一张张自己拼，而且新同事看不懂你那套私有字段。
  4. 第三条最容易被忽略，也是最值钱的：**约定本身是一份别人替你想好的清单**。自己定字段多半只记模型、耗时、token 数三样；约定里还有缓存命中的 token 数、推理 token 数、完成原因、工具调用 id。这些不记不会报错，但等到要回答「上个月缓存省了多少钱」的时候，历史数据里没有就是永远没有了——**遥测的坑是补不回来的**。
  5. 反过来也要说清边界：不是所有字段都塞进公开命名空间。业务自己的维度（任务 id、租户、实验分组）应该放在自己的命名空间里，不要往 gen_ai 前缀底下塞私货，否则升级约定时你分不清哪些是自己的、哪些是人家的。
  6. 可预期的追问是「那约定里没有你要的字段怎么办」。答案是先查一遍确认真的没有，然后放进自己的命名空间，并留意上游有没有在讨论同名字段——真加进约定之后做一次改名，比一直自造要划算。

  Key points

  • Swapping backends changes one export endpoint instead of every instrumentation site and dashboard query.
  • Vendor-provided default views work out of the box instead of being rebuilt chart by chart.
  • The convention is a ready-made field checklist covering cache tokens, reasoning tokens and finish reasons you would not think of.
  • Telemetry gaps are unrecoverable: a field you did not record cannot be reconstructed later.
  • Keep business dimensions under your own namespace rather than inside the public prefix.

  答题要点

  • 换后端时只改导出地址，不用改每一处埋点和每一张面板查询。
  • 各家后端的开箱即用视图直接可用，不必逐张自己拼图。
  • 约定是一份现成的字段清单，缓存 token、推理 token、完成原因这些自己多半想不到。
  • 遥测的坑补不回来：当时没记的字段，事后无法从历史数据里恢复。
  • 业务自有维度放进自己的命名空间，不要塞进公开前缀底下。
• Would you adopt a standard that is still in development and whose attribute names may change? How do you control the upgrade risk?一个仍在开发中、属性名还会改的标准，你会现在就用吗？怎么控制升级风险？

  Common in ChinaCommon overseasDeep dive#observability#opentelemetry#dependency-risk

  How to reason about it · think before answering

  1. On the surface this is about a standard; really it is about how you handle an unstable dependency. Both extremes score poorly: 'wait for stable' forfeits benefits available today, while 'just adopt and fix later' shows you never priced the fix.
  2. Get the facts right first, which alone separates candidates: nothing in the GenAI semantic conventions is marked Stable - everything is Development, there is no 1.0 - and the conventions have moved out of the main semantic-conventions repository into a dedicated one that has not cut a single release tag yet.
  3. Add the most misread detail: the registry pages in the main repository now mark every one of those attributes as deprecated and moved. That is a page relocation, not a deprecation of the attributes. Deleting instrumentation because of that red text has been a common mistake this year; pointing it out shows you read the primary source.
  4. Then give the plan, whose core is shrinking the blast radius: every attribute name appears exactly once, in one constants file, and everywhere else imports it. Add a guard asserting that every emitted name comes from that table, so nobody hand-writes a string - a hand-written typo raises no error, it just silently drops a column from the dashboard.
  5. Second, pin the version into the data itself: freeze the convention version in the constants file and emit it as an attribute on every trace. A year later you can tell which revision a batch of data was recorded under instead of guessing.
  6. Finally, the upgrade move: on a rename, dual-write both names for a transition window, cut dashboards and alerts over to the new one, then drop the old. Expected follow-up - should the rename live in the data pipeline instead? It can, but that moves the debt into the pipeline; dual-writing with a stated removal date is cleaner.

  分析过程 · 先想清楚再作答

  1. 这题问的表面是标准，实际是**你处理不确定依赖的工程习惯**。两个极端答案都拿不到分：「等它稳定了再说」会白白丢掉现在就能拿到的好处，「用，有问题再改」则暴露你没算过改的成本。
  2. 先把事实说准，这一步就能拉开差距：生成式 AI 的那套语义约定至今没有任何条目标记为 Stable，全部处于 Development，没有 1.0；而且它已经从主语义约定仓库搬进了一个独立仓库，那个仓库到现在一个发布 tag 都没打过。
  3. 还要补一句最容易被误读的现状：主仓库的属性登记页上，每一条相关属性现在都带着「已弃用，已移至新仓库」的标记。**那是页面搬家，不是属性被废弃。** 看到红字就把埋点删掉是这一年最常见的误操作，能主动指出这一点，说明你看的是一手页面而不是二手文章。
  4. 然后给方案，核心是**把改动面收敛**：全部属性名只在一个常量文件里出现一次，别处一律引用；再写一条护栏断言所有用到的名字都来自这张常量表，防止有人图省事手写字符串——手写的那个拼错了不会报错，只会在面板上少一列。
  5. 第二件事是把版本钉进数据本身：常量文件里写死约定版本，并把这个版本号作为属性写进每一条链路。一年后翻历史数据时，你能立刻知道那批数据是按哪一版记的，而不是靠猜。
  6. 最后给升级动作：属性改名时做双写过渡（一段时间内新旧名都写），等面板和告警都切到新名再撤掉旧的。可预期的追问是「那要不要在数据管道里做改名映射」——可以，但那是把债转移到了管道上，双写加一个明确的下线日期更干净。

  Key points

  • State the facts: nothing is Stable, everything is Development, and it has moved to a repository with no release tag yet.
  • The deprecation banners in the main repository mean the pages moved, not that the attributes died - do not delete instrumentation over them.
  • Adopt it, but confine changes to a single constants file that everything else imports.
  • Add a guard asserting every emitted attribute name comes from that table, since a hand-typed typo silently drops a column.
  • Pin the convention version in code and emit it as an attribute; on renames dual-write, migrate dashboards, then retire the old name.

  答题要点

  • 先把事实说准：至今无任何条目为 Stable，全部 Development，且已搬进一个尚无发布 tag 的独立仓库。
  • 主仓库页面上的「已弃用」是页面搬家，不是属性被废弃，不能照着它删埋点。
  • 用，但把改动面收敛到一个常量文件，别处一律引用。
  • 加一条护栏断言所有用到的属性名都来自常量表，防止手写字符串拼错后静默少一列。
  • 把约定版本钉进常量文件并作为属性写进链路；改名时双写过渡，切完面板再下线旧名。
• How would you layer the trace tree for one agent run, and what attributes go on each layer?一次 Agent 运行的链路树你会怎么分层？每层各记哪些属性？

  Common in ChinaCommon overseasIntermediate#observability#tracing#span-design

  How to reason about it · think before answering

  1. This tests whether you have actually drawn one. 'Record the model call' earns nothing; give the layers, the operation name per layer, and the rule that decides nesting.
  2. Three layers is the common skeleton. Outermost is the agent invocation, operation invoke_agent, carrying provider, request model, conversation id, plus your own business dimensions in your own namespace. The middle layer is each model turn, operation chat, carrying request parameters, response model and id, finish reasons, and input/output token usage. Innermost is each tool call, operation execute_tool, carrying tool name, tool type, call id, arguments and result.
  3. Operation names cannot be invented. The convention defines an enum of exactly nine values: chat, create_agent, embeddings, execute_tool, generate_content, invoke_agent, invoke_workflow, retrieval, text_completion. An off-enum value is equivalent to not instrumenting at all: backends facet on the enum, and unknown values land in an 'other' bucket that never surfaces in a chart.
  4. Explain the nesting rule: tool spans hang under the model turn that requested them, not as siblings of it, because that is the only way to attribute tool latency to a specific turn. Flatten them in a multi-turn agent and you can no longer say which turn issued the third tool call. Both layouts exist in the wild; what matters is that one company picks one, or hierarchy-based aggregation stops agreeing across services.
  5. Mention two commonly missed fields. Time to first chunk must be stamped when the first streamed chunk arrives and cannot be derived from total duration - for a streaming UI it is the latency the user actually feels. And the evaluation quartet - grader name, score, pass label, explanation - belongs on the root span so the dashboard's failure rate has a correct source.
  6. Finally, payload control: truncate and redact messages, tool arguments and results. Dumping whole conversations into spans is the classic way to blow up storage in one afternoon and the most common path for leaking personal data. Expected follow-up - how do you debug without the full text? Keep it in your own logs and put only a correlating id on the span.

  分析过程 · 先想清楚再作答

  1. 这题考的是「有没有真的画过一棵树」。只说「记一下模型调用」的拿不到分，要给出层级、给出每层的操作名、并解释分层的判据。
  2. 三层是最常用的骨架：最外层是一次 Agent 调用，操作名 invoke_agent，记提供方、请求模型、会话 id，再加上你自己命名空间里的业务维度；中间层是每一轮模型调用，操作名 chat，记请求参数、响应模型与 id、完成原因、输入输出 token 用量；最里层是每一次工具调用，操作名 execute_tool，记工具名、工具类型、调用 id、参数与结果。
  3. 操作名不能自己造。约定里它是一个枚举，一共九个合法取值（chat、create_agent、embeddings、execute_tool、generate_content、invoke_agent、invoke_workflow、retrieval、text_completion）。写一个枚举外的值等于没埋：后端按枚举分面，不认识的值会掉进 other 桶，永远出不了图。
  4. 分层判据要说清楚：工具跨度挂在它所属的那一轮模型调用下面，而不是与模型调用平级。因为工具是那一轮决定要调的，挂进去才能把工具耗时归因到具体某一轮；多轮 Agent 一旦平铺，你就说不清第三次工具调用是第几轮发起的。**两种画法真实世界里都有，重点是全公司统一**，否则按层级做的聚合查询在两个服务之间对不上。
  5. 还要提两个容易漏的字段。一是流式首块延迟，它必须在读到第一个分片时打点，**不能从总耗时推算**，对流式界面来说它才是用户感知的快慢。二是评估结果四件套，把评分器名字、分数、通过标签与理由挂在根跨度上，这样面板上的失败率才有正确来源。
  6. 最后是负载控制：输入输出消息和工具参数结果都要截断并考虑脱敏，整轮对话原样进链路是把存储一次性写爆的经典方式，也是泄漏个人信息最常见的路径。可预期的追问是「那出了问题要看全文怎么办」——把全文留在你自己的日志里，链路只留一个能关联回去的 id。

  Key points

  • Three layers: invoke_agent wraps chat, chat wraps execute_tool, with tools nested under the turn that requested them.
  • Operation names must come from the nine-value enum; anything else lands in the backend's 'other' bucket.
  • Per layer: root carries provider, model and conversation id; chat carries request parameters, response id and token usage; tool spans carry name, type, call id, arguments and result.
  • Time to first chunk must be stamped on arrival of the first chunk, never derived; the evaluation quartet goes on the root span.
  • Truncate and redact messages and tool payloads; keep full text in logs and put only a correlating id on the span.

  答题要点

  • 三层：invoke_agent 包 chat，chat 包 execute_tool；工具挂在发起它的那一轮模型调用下面。
  • 操作名只能取九个合法值之一，自造值会掉进后端的 other 桶，等于没埋。
  • 各层属性：根层记提供方、模型与会话 id；chat 层记请求参数、响应 id 与 token 用量；工具层记工具名、类型、调用 id、参数与结果。
  • 首块延迟必须在第一个分片到达时打点，不能从总耗时推算；评估四件套挂在根跨度上。
  • 消息与工具参数结果要截断并脱敏，全文留在日志里、链路只放关联 id。
• Recording every trace in production is too expensive. How do you design sampling so that you do not throw away the failures you actually need?线上全量记录太贵，你怎么设计采样策略才不会把真正的故障样本丢掉？

  Common in ChinaCommon overseasDeep dive#observability#sampling#production

  How to reason about it · think before answering

  1. The trap is that the word 'sampling' makes people reach for a random percentage drop. The real question is when the decision is made.
  2. Head-based sampling rolls the dice as the request arrives. It is cheap and simple, but the cost is fatal: the decision happens before you know whether this trace will fail. A failure mode that hits 1% of requests is discarded 99% of the time before it goes wrong - the samples you most need are precisely the ones most likely to be dropped.
  3. Tail-based sampling is the right shape: wait until the trace finishes and the outcome is known, then decide. The rules can be blunt - keep every failure, keep everything slower than p99, keep a small proportion of fast successes as a baseline. The baseline matters: keep only failures and you cannot compute a failure rate or see what healthy looks like.
  4. For agents, say explicitly what 'failure' means: not the HTTP status code. A request that refunded the wrong amount still returns 200. The signal should come from the evaluation attributes - put the grader's pass label on the root span and let the sampling rule read it. That is a direct payoff of stitching offline evaluation into online telemetry.
  5. One implementation detail separates people who have done this: use a stable hash of the trace id, not a random number. Randomness keeps some spans of a trace and drops others, producing a truncated tree that is worse than nothing - it makes a stage look as if it never happened.
  6. Name the cost too: tail sampling must buffer all spans of a trace until the verdict is known, so the collector has to survive memory pressure and out-of-order arrival, and long traces need a timeout that forces a flush. Expected follow-up - how do you keep spend bounded? Budget the post-sampling write volume and, when over budget, lower the retention of successful traces first; the retention of failures and slow traces never moves.

  分析过程 · 先想清楚再作答

  1. 这题的陷阱在于「采样」这个词会让人下意识想到按比例随机丢。真正的考点是**在什么时刻做这个决定**。
  2. 头部采样是请求一进来就掷骰子决定记不记。它便宜、实现简单，但代价是致命的：这个决定是在你还不知道这条链路会不会出问题的时候做的。于是一条命中率 1% 的故障链路，有 99% 的概率在它出问题之前就已经被丢掉了——**线上最需要的那批样本，恰好是最容易被采样掉的那批**。
  3. 正确的做法是尾部采样：等链路跑完、结果已知，再决定留不留。规则可以写得很直白——失败的一条不落，超过 p99 的慢链路全留，成功且不慢的按比例留一小部分当基线。留基线很重要，全丢掉的话你手里只剩故障样本，没法算失败率，也看不出正常态是什么样。
  4. 对 Agent 来说「失败」的判据要特意说清楚：不是 HTTP 状态码。一个退错了钱的请求照样是 200。判据应该来自评估结果属性——把评分器的通过标签挂在根跨度上，采样规则直接读它。这也是离线评估与线上监控缝在一起之后的直接收益。
  5. 还有一个实现细节能看出有没有真做过：决定留不留要用 traceId 的稳定哈希，不要用随机数。随机数会让一条链路的一部分跨度被留下、另一部分被丢掉，拼出来是一棵残树，比完全没有更糟——它会让人以为某一段根本没发生。
  6. 代价也要主动说：尾部采样必须先把一条链路的全部跨度缓存到能判定结果为止，所以收集端要扛住内存与乱序到达，长链路还要设超时强制出清。可预期的追问是「那怎么保证成本可控」——给采样后的写入量设预算，超预算时先降成功样本的留存比例，**失败与慢链路的留存率永远不动**。

  Key points

  • The question is not how much to drop but when to decide: head-based sampling rolls the dice before the outcome is known.
  • Its fatal flaw: a low-frequency failure mode is almost always discarded before it goes wrong.
  • Tail-based rules: keep all failures, keep everything above p99 latency, keep a small sampled baseline of fast successes.
  • Failure for an agent comes from evaluation attributes, not HTTP status - a wrong refund still returns 200.
  • Use a stable hash of the trace id, never a random number, or you get truncated trees; under budget pressure lower only the retention of successful traces.

  答题要点

  • 关键不是丢多少，而是在什么时刻决定：头部采样在结果未知时就掷骰子。
  • 头部采样的致命问题：低命中率的故障链路极大概率在出问题之前已被丢掉。
  • 尾部采样规则：失败全留、超 p99 的慢链路全留、成功且不慢的按比例留一小部分当基线。
  • Agent 的失败判据来自评估属性而不是 HTTP 状态码，退错钱的请求同样返回 200。
  • 用 traceId 的稳定哈希而不是随机数，否则会产出残缺的链路树；预算紧张时只降成功样本留存率。

Comments