Week 1 · D7About 4 hours

Examining the Suite Itself: Saturation, Broken Tasks and Grading Defects

The last day turns the lens on the evaluation system. A suite approaching a perfect score has stopped carrying information, and a single mis-written grader can make a good model look fifty points worse. Today you health-check the suite, establish the discipline of reading transcripts, and assemble seven days of work into a portfolio project.

Today's goals 0/3

Recognize a saturated suite and say what to do next instead of continuing to report that flattering score
Separate model problems from grading problems in a batch of failures, and name the typical shapes a grading defect takes
Explain which class of gap each of the five non-automated quality practices covers

Today's Goals

Recognize a saturated suite and say what to do next instead of continuing to report that flattering score
Separate model problems from grading problems in a batch of failures, and name the typical shapes a grading defect takes
Explain which class of gap each of the five non-automated quality practices covers

Plain-Language Walkthrough

The assumption six days have been resting on

Over six days you built a complete evaluation system: it produces two probabilities, it runs against a twenty-four task benchmark set covering both directions, its judge has been calibrated for agreement, it catches detours and loops, every run carries a trace and a cost, and the pipeline turns red when quality drops.

Through all six days, one assumption went unchallenged: the suite is correct. If the score dropped, the agent is the problem.

Today is about challenging it.

An evaluation system is code, and code has bugs. Bugs in an evaluation system have one especially nasty property: they do not crash your program. They hand you a wrong conclusion, and then you take that conclusion and go fix something that was never broken.

A fifty-three point lesson

Start with a real case, because it is the most persuasive argument this course can offer.

A model was evaluated on a public benchmark and scored 42. That reads as a capability problem.

Investigation found the problem was in the grading. That benchmark compared numeric answers with strict string equality. The model answered "96.12", the reference was "96.124991…", the two strings are not equal, marked wrong. Alongside it: ambiguous task descriptions, and randomized tasks that could not be reproduced.

Once the grading was fixed, the same model on the same questions scored 95.

A gap of fifty-three points, entirely produced by defects in the evaluation and entirely unrelated to model capability.

If nobody had investigated, the conclusion would have propagated: this model is not good enough, let us swap it out. The team would have spent weeks doing something completely unnecessary.

Today's lab ships a minimal reproduction of that case. Same agent, same refund, identical amount, two graders attached:

TextText

-- what a grading defect costs (same agent, same action, two graders) --
  strict equality comparison: 0%    integer domain comparison: 100%
  the refund amount is identical. The whole gap comes from three lines in the grader.

Convert 4990 minor units into 49.9 with one arithmetic step that can introduce error and you get 49.89999999999999. Which is not equal to 49.9.

Suite saturation: the score is still there, the information is gone

The second problem is subtler, because its symptom is a very good-looking score.

A freshly built suite might pass thirty percent of the time, and every improvement produces a visible jump. Six months later it sits at 98 percent.

It still has value, but the value changed: only regression value is left. It no longer carries an improvement signal.

When a score moves from 98 to 98.5, you cannot tell progress from noise. And a genuine leap in capability might show up on that suite as a single point.

There is a mirror-image case worth remembering. A team evaluated a new model with one-shot single-turn tests and concluded the improvement was marginal. When they rebuilt the evaluation to be multi-step with real tools, the model's actual gains on long tasks became visible. The model had improved. Their ruler could not measure that dimension.

Three shapes of a broken task

The third problem is a task that is simply written wrong. D2 covered validating a reference solution; today fills in how to recognize the rest.

Shape	Symptom	How to find it
Ambiguous description	Pass rate hovers in the middle, and reading transcripts shows the agent doing a different but reasonable thing	Have two people judge independently; they disagree
Unverifiable criterion	The criterion says something like "answers professionally", which nothing can judge automatically	You discover you cannot judge it either while writing the reference solution
Irreproducible randomness	The same input has a different correct answer each time, and the grader is comparing against a fixed one	Run twice with a fixed seed; the results differ

Plus the rule from D2: a task with a long-running zero pass rate should be suspected before the agent is. A genuinely hard but solvable task usually gets lucky at least once over many trials. An unbroken zero is more like a wall than a slope.

Today's health checker automates these. But its boundary has to be stated plainly: it can only find the categories it knows how to look for. A report with no findings is not the same as a healthy suite.

A detector that is always true is worse than no detector

While writing today's health checker I walked straight into a trap, and it is worth telling exactly as it happened.

The "baseline representativeness" check answers the question: was your baseline number recorded on an unusually lucky run? If it was, every normal run afterwards will look like a regression, and D6's gate will produce false alarms indefinitely.

The first version of the criterion read: warn if the baseline is greater than or equal to the highest value in the distribution. The report it produced said:

TextText

x unrepresentative-baseline [suite level]
    baseline 60.0% sits at the top of 12 same-configuration runs (range 60.0%-60.0%, median 60.0%)

The range is 60.0 percent to 60.0 percent — that batch of tasks does not move at all, and twelve runs scored identically. When that happens, the baseline is necessarily both the maximum and the minimum, so "sits at the top" is always true. The detector therefore warns on every perfectly stable suite, and the warning text reads perfectly convincing while doing it.

This check also answers the question D6 left open. D6 taught how to record a baseline and how to set a threshold, but never said which run the baseline should be recorded on. The answer: a representative one, and you need a way to verify that it actually is.

Agents exploit loopholes, and you left the loopholes there

One more phenomenon, infuriating the first time you meet it: the agent finds a shortcut you never imagined, "completes" the task, and does it in a way you did not want at all.

It discovers that editing the test file makes the tests pass. It finds the answer in the git history.

Before getting angry, consider one thing: those loopholes were left there by the evaluation designer. If the criterion is "the tests pass", then editing the tests does satisfy the criterion. The agent is optimizing the objective function you wrote down. It did not cheat; the objective was written imprecisely.

The productive response is to treat every exploit as a free audit of your evaluation. It points, with precision, at the gap between your criteria and your actual intent.

Reading transcripts is a discipline, not a suggestion

The last idea of the day, and the one from this course most worth keeping:

Do not believe any evaluation score unless somebody has actually read a few transcripts.

The groundwork is already laid. Grading can be wrong, a task can be ambiguous, an agent can exploit a loophole, a detector can be vacuously true. None of those four is exposed by the score itself. Every one of them requires a person to open a specific run and look.

The trouble is that "read more transcripts" written in a document is the same as not writing it. So today it becomes a function: the end of the health report prints a list of which transcripts to read this round and why.

The sampling rule is deliberately not random. Failures go in, because their information density is highest. A few passing trials go in too — looking only at failures builds the illusion that the agent gets nothing right, and a criterion that is too lenient is a problem you can only see inside a passing sample.

Swiss cheese: no single layer holds

Finally, pull back. Automated evaluation is one layer of a quality system, and it has places it cannot see.

Practice	Which class of gap it covers	Cost
Automated evaluation	Regression on known problems, runnable on every commit	Only covers scenarios you thought of
Production monitoring	Real user behavior, cases no synthetic task anticipated	Only after the fact, low signal-to-noise
Controlled experiments	Real outcomes, confounders excluded	Slow, only measures what already shipped
User feedback	Problems you did not anticipate at all	Sparse, skewed toward severe issues
Humans reading transcripts	Subtle quality that automated checks miss	Time-consuming, unevenly distributed
Systematic human studies	The gold standard for subjective tasks, used to calibrate model judges	Most expensive and slowest

Every layer has holes, but the holes are in different places. Stacked, what one layer misses the next one catches.

Do not expect any single layer to hold quality on its own — including the one you built over these seven days.

Source Reading

Today returns to the second half of the Anthropic post; the first six days lived in its first half. The back half is about maintaining evaluations over the long run: the discipline of reading transcripts, evaluation saturation, how to organize a team that keeps evaluations alive, and that six-row table of quality practices. One idea there is worth noting — eval-driven development: write the evaluation before the agent has the capability, and let the evaluation define what you intend to build. That is the same idea as test-driven development.

Harbor is the other source, a containerized agent evaluation framework. Read it for two things: how it pushes "a clean environment per trial" all the way down to the container level (these seven days did it in-process with object copies, which is fine for teaching and not enough for production), and how it plugs different agents into a single task interface.

Checking a license means opening the LICENSE file in the repository, not reading the label on the hosting platform's page — that field is auto-detected, it reports "not specified" when detection fails, and the projects it fails on are precisely the ones that made special arrangements. Whether you build or adopt depends on whether your team needs to self-host, whether you will redistribute, and whether you could take over the project if it stopped being maintained.

Hands-On Lab

🧪 D7: Suite Health and Portfolio Wrap-Up

Code location: labs/agent-evals-7days/day-07-suite-health

Today's lab merges every module from the seven days into one project, then adds health checking and reporting.

Four exercises map onto four classes of problem: saturation, long-running zeros, grading defects, and baseline representativeness. The fourth is the important one — it is the easiest place to write a vacuously true implementation, and the README covers it separately.

The core shape of the health checker looks like this. Note that every finding carries a next action, not just a complaint:

suite-health.js

export function detectUnrepresentativeBaseline(baselineRate, noiseRates) {
  if (noiseRates.length < 5) return []
  const sorted = [...noiseRates].sort((a, b) => a - b)
  const lo = sorted[0]
  const hi = sorted[sorted.length - 1]
 
  // The critical line: stay silent when the distribution has not spread.
  // When every run scores the same, the baseline is necessarily both the
  // maximum and the minimum, so "sits at the top" is always true and the
  // detector would warn on every stable suite.
  if (hi - lo < 1e-9) return []
 
  if (baselineRate >= hi) {
    return [{ kind: 'unrepresentative-baseline', severity: 'error',
      detail: 'baseline recorded on a lucky run; every normal run will look like a regression and the gate will keep crying wolf' }]
  }
  return []
}

suite_health.py

def detect_unrepresentative_baseline(baseline_rate, noise_rates):
    if len(noise_rates) < 5:
        return []
    sorted_rates = sorted(noise_rates)
    lo, hi = sorted_rates[0], sorted_rates[-1]
 
    # The critical line: stay silent when the distribution has not spread.
    # When every run scores the same, the baseline is necessarily both the
    # maximum and the minimum, so "sits at the top" is always true and the
    # detector would warn on every stable suite.
    if hi - lo < 1e-9:
        return []
 
    if baseline_rate >= hi:
        return [{
            "kind": "unrepresentative-baseline",
            "severity": "error",
            "detail": "baseline recorded on a lucky run; every normal run will look like a regression and the gate will keep crying wolf",
        }]
    return []

The manual checklist contains a hand-run mutation test for exercise four: comment out the one task that actually moves and run again, and the baseline check should stay silent; restore it, and the warning comes back. Only after verifying both directions do you know the detector works.

Interview Questions

Today's four questions close out the course: what to do once a suite saturates, how to tell a model regression from a mis-written grader, why reading transcripts is a discipline, and how to choose an evaluation framework.

The fourth follows up on licensing. That is where people tend to get exposed — plenty of candidates can name frameworks and cannot answer "does this project's license let us self-host and redistribute". Which happens to be a question no real adoption decision gets to skip.

Checklist and Tomorrow

By the end of the seven days you should be able to:

State the criterion for suite saturation, and what to do once it saturates (which is not deleting the task)
Name the three typical shapes of a grading defect, and explain why money is compared as an integer
Separate model problems from grading problems in a batch of failures, and give the investigation order
Explain why a detector that is always true is worse than no detector
Say which class of gap each of the six quality practices covers
Get all nine assertions green with MOCK=1 pnpm selftest
Run the manual mutation test yourself: comment out the task that moves, watch the detector go silent; restore it, watch it come back

That is the end of the seven days. What you hold now is a complete evalkit: it produces two probability metrics, it carries a benchmark set covering both directions, its judge has been calibrated for agreement, it catches detours and loops, every run has a trace and a cost, it turns the pipeline red on a regression, and it can examine its own health.

Pointing it at your own agent is the real homework for this course. Of the frozen files, only the target is course-specific; swapping in your own agent means implementing a single interface: given a task and a world, return a transcript.

One closing thought. Every method taught here serves one very plain purpose: to let you answer honestly whether the thing actually works — including when the answer is no. An evaluation system that only reports good news is more dangerous than having no evaluation system at all.

Interview questions

Your evaluation suite is passing at 98 percent. What do you do next?你们的评估套件通过率已经百分之九十八了，接下来你会做什么？
Common in ChinaCommon overseasIntermediate#evaluation#saturation#suite-health
How to reason about it · think before answering
This tests whether you recognize saturation. Answering 'great, the system is high quality' admits you have never thought of an evaluation suite as having a lifecycle.
State what 98 percent means: this suite has exhausted its improvement signal. Moving from 98 to 98.5 is indistinguishable from noise, and a genuine capability jump might show up as a single point. All that remains is regression value.
Then give actions, and the first one is not deleting tasks: move the saturated ones into the regression set, where they keep guarding against backsliding. Deleting them forfeits regression protection for that capability, which is the most common mishandling.
Second, write harder tasks, and say where they come from: recent production incidents, user complaints, scenarios the current agent clearly handles badly but the business genuinely needs. A new capability set should start at a low pass rate, because that is what leaves room to climb.
Third, check that the ruler measures the right dimension at all. A real counterexample: a team evaluated a new model with one-shot single-turn evals, concluded the gains were unremarkable, and only saw the real benefit after building a multi-step evaluation with real tools. The model had improved; the ruler could not see that dimension.
Expected follow-up: how do you know the 98 percent is real rather than a loose criterion? Sample the transcripts of passing trials - looking only at failures can never surface a criterion that is too lenient.
分析过程 · 先想清楚再作答
这题考的是「能不能识别饱和」。回答「很好，说明系统质量高」的，等于承认自己从没想过评估也有生命周期。
先说清楚 98% 意味着什么：这个套件的**改进信号已经耗尽**。分数从 98 涨到 98.5，你分不清是真进步还是噪声；而一次真正的能力飞跃，在这个套件上可能只体现为一个百分点。它现在只剩回归价值。
然后给动作，而且第一个动作不是删题：**把饱和的任务移进回归集**，它们继续守着「不要退步」这条线。删掉等于放弃了对这项能力的回归保护，这是最常见的错误处置。
第二个动作是出更难的新题，而新题从哪来要说得出来：从最近的线上故障、从用户抱怨、从当前 Agent 明显做不好但业务真的需要的场景。新的能力集应该**从低通过率起步**，那才是有爬坡空间的尺子。
第三个动作是检查尺子本身量的维度对不对。有个真实的反例：某团队用一次性的单轮评估测新模型，结论是提升不明显；改成多步、带真实工具的评估之后，才看到它在长任务上的真实收益。**不是模型没进步，是尺子量不到那个维度。**
可预期的追问是「那你怎么知道 98% 是真的，不是判据太松」。答案是抽样读通过样本的轨迹——只看失败样本永远发现不了判据太松这类问题。
Key points
98 percent means the improvement signal is exhausted; small deltas are indistinguishable from noise.
First action is moving saturated tasks into the regression set, not deleting them.
Write harder tasks sourced from incidents, complaints, and business-critical weak spots.
A new capability set should start at a low pass rate to leave room to climb.
Verify the ruler measures the right dimension - single-turn evals miss multi-step gains.
答题要点
98% 说明改进信号已耗尽，只剩回归价值，小幅变化无法与噪声区分。
第一个动作是把饱和任务移入回归集，**不是删掉**——删掉会放弃回归保护。
出更难的新题，来源是线上故障、用户抱怨、业务需要但当前做不好的场景。
新能力集应从低通过率起步，才有爬坡空间。
检查尺子量的维度对不对：单轮评估可能量不到多步能力的进步。
A model suddenly drops a lot of points on your suite. How do you tell a real regression from a broken grader?一个模型在你的套件上突然掉了很多分，你怎么区分是模型退化还是判分写错了？
Common in ChinaCommon overseasDeep dive#evaluation#debugging#grading-defect
How to reason about it · think before answering
This tests your debugging order. Saying 'roll back the model' loses half the credit - you have not yet shown the problem is on the model side.
First look at the shape of the drop. A genuine model regression is usually diffuse: everything sags a little. A grading defect is typically concentrated: one class of tasks goes to zero while the rest is untouched. The shape alone is a strong signal.
Second, read the failure reasons. Grading defects have a recognizable fingerprint: two numbers differing in the last digits yet judged unequal (strict equality, especially after a float conversion), case or whitespace mismatches, and the 'expected X, got X' pattern that reads identical yet fails. In one public case a model scored 42 because '96.12' did not equal '96.124991...', and the same model on the same suite scored 95 once grading was fixed - fifty-three points entirely from the evaluation.
Third, validate with a reference solution: feed a hand-verified correct answer to the grader. If the reference solution fails, the fault is one hundred percent in the evaluation. This is the cleanest test.
Fourth, check whether the environment moved: dependency upgrades, an upstream API changing its response shape, test data someone edited. These masquerade as model regressions and cause the most finger-pointing.
Only then read transcripts to confirm the model really did worse. The order runs from the evaluation side toward the model side, because evaluation-side faults are both easier to check and more common. Expected follow-up: how do you make this faster? Record a complete environment fingerprint in the baseline - model version, prompt version, task-set version, random seed. Every missing field is one more variable you cannot rule out.
分析过程 · 先想清楚再作答
这题考排查顺序。直接说「回滚模型」的丢一半分——你还没证明问题在模型那边。
第一步是看**掉分的形状**。真实的模型退化通常是弥散的：各类任务普遍降一点。判分缺陷往往是**集中的**：某一类任务集体归零，而其他类纹丝不动。形状本身就是很强的线索。
第二步读失败理由。判分缺陷有很典型的指纹：两个数字只差一点点却判不相等（严格相等比较，尤其是浮点换算之后）、大小写或空格差异、以及「期望 X 实际 X」这种看起来一模一样却判失败的情形。有个公开案例，某模型因为「96.12」不等于「96.124991…」初评 42 分，判分修好后同一套题得 95 分——**五十三个百分点全部来自评估缺陷**。
第三步用参考解验证：把一条人工做对的答案喂给评分器。参考解都过不了，那 100% 是评估的问题，与模型无关。这是最干净的判据。
第四步检查环境是否变了：依赖升级、上游 API 改了返回格式、测试数据被人动过。这些会伪装成模型退化，而且在团队里最容易互相甩锅。
最后才是读轨迹确认模型确实做错了。整个顺序是**从评估侧走向模型侧**，因为评估侧的问题更容易排查、也更常见。可预期的追问是「怎么让这个排查变快」——答案是基线快照里记全环境指纹（模型版本、提示词版本、任务集版本、随机种子），少记一样就多一个没法排除的变量。
Key points
Check the shape: real regressions are diffuse, grading defects are concentrated.
Read failure reasons for grading fingerprints: strict equality, float conversion, case and whitespace.
Validate with a reference solution - if a known-correct answer fails, the fault is in the evaluation.
Check environment changes: dependencies, upstream response shapes, edited fixtures.
Debug evaluation-side first; a complete environment fingerprint makes variables ruleable-out.
答题要点
先看掉分形状：真实退化弥散，判分缺陷集中在某一类任务。
读失败理由找判分指纹：严格相等、浮点换算、大小写与空格差异。
用参考解验证：人工做对的答案过不了评分器，就一定是评估的问题。
检查环境变更：依赖升级、上游返回格式变化、测试数据被改。
排查顺序从评估侧到模型侧；基线快照记全环境指纹才能快速排除变量。
Why should you not trust an evaluation score before someone has read transcripts? And how do you make transcript reading a team habit?为什么说没读过轨迹就不该相信评估分数？你会怎么把读轨迹变成团队习惯？
Common in ChinaCommon overseasIntermediate#evaluation#transcripts#process
How to reason about it · think before answering
The first half asks why, the second asks how. Answering only the first scores poorly - 'we should read more transcripts' carries no information, everyone agrees with it, and nobody does it.
Be concrete about what a score cannot show: the grader may be wrong (the score is still a number), the task may be ambiguous (the agent solved a different reasonable problem), the agent may have exploited the criterion (editing tests so tests pass), the detector itself may be vacuous (always firing or never firing). None of these four surface in the score; all of them are visible in a specific run.
Call out the exploitation case, because the attribution is easy to get backwards. An agent finding a shortcut is not cheating - it is evidence of a gap between your criterion and your actual intent. Every exploit is a free audit that points at exactly where the gap is.
For adoption, the key move is turning the practice into an artifact. Have the report emit a reading list for this round, with a reason attached to each entry. People tick off a list; they do not act on an exhortation.
Spell out the sampling rule, and it must not be purely random: failures always go in because they carry the most information, and a few passing trials go in too - looking only at failures breeds the illusion that nothing works, and a criterion that is too lenient is visible only among passes.
Expected follow-up: how many is enough? Honestly there is no universal number, but there is an operational test: once you start seeing the same failure cause repeat, the marginal value of this round has dropped. Also mandate a re-read after any change to graders, model, or task set, because that is exactly when new problems appear.
分析过程 · 先想清楚再作答
这题前半问原因，后半问落地。只答前半段拿不到高分——「应该多读轨迹」这句话本身毫无信息量，所有人都同意，然后没有人做。
原因要具体列出分数看不见的东西：判分可能写错（分数照样是个数）、任务描述可能有歧义（Agent 做了另一件合理的事）、Agent 可能钻了判据的空子（改测试让测试通过）、检测器本身可能恒真（永远报警或永远沉默）。**这四类没有一类会体现在分数上**，它们只在具体的运行记录里看得见。
特别要提钻空子这一类，因为它的归因容易搞反：Agent 找到捷径不是它作弊，是**你的判据和你的真实意图之间有差距**。每一次钻空子都是一次免费的评估审计，精确指出了差距在哪。
落地部分的关键是**把它从建议变成产物**。做法：让评估报告自动输出一份「本轮该读的轨迹」清单，附上每条为什么被选中。人对着清单打勾，比对着一句倡议要可执行得多。
抽样规则要说清楚，而且不能是纯随机：失败样本必须进（信息密度最高），**通过样本也要抽几条**——只看失败会形成「它什么都做不对」的错觉，而且判据太松这类问题**只有在通过样本里才看得见**。
可预期的追问是「读多少条才够」。诚实的答案是没有普适数字，但可以给操作性判据：读到你开始重复看到同一类失败原因，这一轮的边际收益就下来了。另外每次改判据、换模型、加新任务之后必须重读，因为那正是新问题最可能出现的时候。
Key points
Scores hide four failure classes: broken grading, ambiguous tasks, criterion exploitation, vacuous detectors.
An exploit reveals a gap between criterion and intent - it is a free audit, not cheating.
Make it an artifact: have the report emit a reading list with a reason per entry.
Sampling is not random: always include failures, and include some passes, since a lenient criterion is only visible there.
Mandate a re-read after any grader, model, or task-set change.
答题要点
分数看不见四类问题：判分写错、任务有歧义、Agent 钻空子、检测器恒真。
Agent 钻空子说明判据与真实意图有差距，是免费的评估审计，不是作弊。
落地要把它变成产物：评估报告自动输出「本轮该读的轨迹」清单并附选中理由。
抽样不能纯随机：失败样本必进，通过样本也要抽——判据太松只在通过样本里可见。
改判据、换模型、加新任务之后必须重读，那是新问题最可能出现的时候。
Build your own evaluation framework or adopt an existing one? Give your decision criteria, and say what you check in the license.自建评估框架还是用现成的？说出你的判断依据，以及选型时会看许可的哪些方面。
Common in ChinaCommon overseasDeep dive#evaluation#tooling#licensing
How to reason about it · think before answering
Two things are being tested and most people only answer the first. The front half is engineering judgment; the back half is licensing awareness, which is unavoidable in real procurement and where candidates most often come up short.
For the engineering half, give criteria rather than a verdict: how unusual your tasks and criteria are (the more bespoke the business rule, the costlier the adaptation), whether you need self-hosting, whether anyone will maintain it, and the decisive one - are you buying a framework or a platform? The hard part of evaluation was never the code that loops over test cases; it is high-quality tasks and graders, and you are writing those yourself regardless. Framework choice matters less than most people assume.
The sensible compromise: adopt something for running batches and rendering reports, write tasks and graders yourself. Pick a framework quickly and spend the saved effort on test-case quality.
For licensing, say what you check: whether you may self-host; whether you may offer it to third parties as a service, which disqualifies a good number of source-available projects; whether different directories carry different terms (an open core with a separately licensed enterprise directory is a common structure); and whether any copyleft term reaches your proprietary code.
Stress one operational detail: read the LICENSE file in the repository, not the badge on the hosting platform. That field is auto-detected and reports 'not specified' whenever detection fails - and detection fails precisely on the projects with unusual arrangements. Three verified examples: an observability platform widely described as MIT has LICENSE text carving out three enterprise directories; an evaluation platform widely called open source ships under Elastic License 2.0, which explicitly forbids offering it as a hosted service and is not OSI-approved; and a framework commonly attributed to a model vendor actually comes from a different organization.
Expected follow-up: what if the project stops being maintained? Make that a selection criterion too - does the license let you fork and maintain it, how many organizations depend on it, can your team read the core logic? A dependency you cannot take over is a dependency that will eventually block you.
分析过程 · 先想清楚再作答
这题有两个考点，很多人只答得出第一个。前半是工程判断，后半是法务意识——后者恰恰是真实选型里绕不过去的一关，也是最容易露怯的地方。
工程判断部分先给判据而不是结论：任务与判据的特殊程度（业务判据越独特，现成框架的适配成本越高）、要不要自托管、团队有没有人维护、以及最关键的一条——**你要的是框架还是平台**。评估的难点从来不在跑测试用例的那段代码，而在**高质量的任务和评分器**，那部分无论如何都得自己写。所以框架选型的权重其实没有多数人以为的那么高。
合理的折中是：跑批与报告用现成的，任务与评分器自己写。先快速选一个框架，把精力压在测试用例质量上。
许可部分要说清楚看什么：① **能不能自托管**；② **能不能作为服务提供给第三方**——这一条会卡住相当一部分「源码可见」的项目；③ 有没有**分目录的差异化许可**（核心开源、企业版目录另有条款是常见结构）；④ 传染性条款会不会影响你的闭源部分。
特别要强调一个操作细节：**看许可要落到仓库里的 LICENSE 文件，不要只看代码托管平台页面上那个标签**。那个字段是自动识别的，识别不出来就报「未指定」，而它识别不出来的恰恰是那些做了特殊安排的项目。实测过三个例子：一个被广泛称作 MIT 开源的可观测平台，LICENSE 原文写明有三个企业版目录另有许可；一个被普遍称作开源的评估平台用的是 Elastic License 2.0，明文禁止作为托管服务提供给第三方，并不是 OSI 认可的开源；还有一个常被误认为出自某家模型厂商的框架，其实来自另一个机构。
可预期的追问是「这个项目停止维护了怎么办」。答案是把它当作选型判据之一：许可允许你 fork 并自行维护吗？有多少组织在依赖它？核心逻辑你的团队读得懂吗？一个你接不了手的依赖，就是一个将来会卡住你的依赖。
Key points
Criteria: how bespoke your rules are, self-hosting needs, maintenance ownership, framework versus platform.
The hard part is tasks and graders, which you write either way, so framework choice carries less weight.
License checks: self-hosting, offering as a service, per-directory terms, copyleft reach.
Read the repository's LICENSE file - the auto-detected badge is often wrong on projects with unusual terms.
Treat abandonment as a criterion: can you fork it, who else depends on it, can your team take it over?
答题要点
判据：判据的特殊程度、是否要自托管、有没有人维护、要的是框架还是平台。
评估的难点在任务与评分器，那部分必须自己写，所以框架选型权重没那么高。
许可看四件事：能否自托管、能否作为服务提供给第三方、是否分目录差异化、有无传染性。
**必须读仓库里的 LICENSE 文件**，平台上那个自动识别的标签在特殊安排的项目上经常不准。
把「停止维护怎么办」纳入选型：许可是否允许 fork、依赖它的组织多不多、团队能否接手。