Evaluation First: Building a Golden Set, Computing Recall and Ranking Metrics, Using a Model as Judge for Faithfulness

Building a golden set: writing questions backwards from the corpus

The direction for constructing a golden set is the opposite of most people's instinct. The instinct is to imagine what users will ask and then check whether the corpus has it; the correct approach is the reverse: open the corpus and read out of each document what it can answer.

The reason is simple: the first approach produces a pile of questions whose answers you do not know, leaving nowhere to start when labeling; the second fixes the answer document at the moment you write the question. The lab's 20 questions came out that way, with every answerDocIds being the document I was reading while writing it.

One entry looks like this:

{
  "id": "q17",
  "question": "Temporarily raising a workspace's trace sampling rate to 100% is what change class? How many working days in advance must the ticket be filed under the standard?",
  "answerDocIds": ["doc-022", "doc-021"],
  "type": "multi"
}

That type field decides how a question counts as hit, in three classes, and missing one leaves your scale blind:

• single: one document answers it. With several answer documents listed, any one entering the context counts as a hit.
• multi: multi-hop, requiring all answer documents in the context to count. Gathering half is the same as not answering, and counting it as half a hit is a consolation prize you awarded yourself.
• none: the corpus contains no answer. It does not enter recall; it tests whether the system stays quiet.

The third is the most easily skipped and the least skippable. The reason is this: a system that only ever answers scores near full marks on an evaluation set containing only answerable questions — it stuffs material in every time and the model invents a passage every time, and your evaluation set has no should-have-refused column, so that fatal flaw is entirely invisible in the report. Unanswerable questions are the only thing that makes fabrication visible.

Writing unanswerable questions has its craft too: strong distractor terms are mandatory. The lab asks which browsers the web client supports and what minimum versions are required, and the corpus happens to contain "please include your browser and version when filing a ticket"; it asks how much CPU and memory an on-premises deployment needs, and the corpus contains both "on-premises deployment" and "CPU and memory" while never stating hardware specifications. An unanswerable question with no distractors is too easy — the retriever fetches nothing and what you measured was the tokenizer rather than the system.

For coverage, a ratio you can copy directly: start at 20 questions, with at least 3 multi-hop and at least 3 unanswerable, and the rest single-document. Twenty is enough to separate unusable from broadly usable; a production system wants 100 to 200, with every production incident's question added to it — an evaluation set is grown.

Three retrieval metrics, each sensitive to a different failure

Only three metrics are computed on the retrieval side. They are not three flavors but three layers of different failures.

Recall asks whether the answer entered the context. Note the criterion: entering the context is not ranking in the top 5 but genuinely fitting inside a 600-token context budget. Comparing by a fixed count is unfair — large chunks fit two thousand-odd tokens in five and small ones only a few hundred, and the winner merely bought more goods. That criterion matches D4 exactly, so the two days' numbers can sit together.

Mean reciprocal rank asks where the first relevant result ranks. First place scores 1, second 0.5, tenth 0.1, none 0, averaged over all questions. It is extremely sensitive to the answer still being present and merely pushed back — something recall cannot see at all until the budget bursts.

Normalized discounted cumulative gain asks how good the whole top ten is. Each position's gain is discounted by rank, discounted harder further down, then divided by an ideal ordering's score to normalize into 0 to 1. Its difference from reciprocal rank: reciprocal rank recognizes only the first result, and one hit plus nine pieces of garbage looks identical to five hits in its eyes, while this metric counts every one of the top ten. Reranking optimizes exactly this one.

The three implementations are all in the lab's metrics.ts, and the core is only this long:

// Recall: single hits when any document enters the context, multi requires all of them
export function isHit(t) {
  if (t.type === 'none') return false
  if (t.type === 'multi') return t.answerDocIds.every((id) => t.contextDocIds.includes(id))
  return t.answerDocIds.some((id) => t.contextDocIds.includes(id))
}
 
// Reciprocal rank: the reciprocal of the first relevant result's rank.
// Indexes start at 0 and ranks at 1; being one off makes every score too high
export function reciprocalRank(t) {
  const rank = t.rankedDocIds.findIndex((id) => t.answerDocIds.includes(id))
  return rank === -1 ? 0 : 1 / (rank + 1)
}
 
// nDCG: each of the top k discounted by 1 / log2(rank + 1), divided by the ideal ordering's score
export function ndcgAt(t, k = 10) {
  const gains = t.rankedDocIds.slice(0, k).map((id) => (t.answerDocIds.includes(id) ? 1 : 0))
  const dcg = gains.reduce((sum, g, i) => sum + g / Math.log2(i + 2), 0)
  let idcg = 0
  for (let i = 0; i < Math.min(t.relevantChunkCount, k); i += 1) idcg += 1 / Math.log2(i + 2)
  return idcg === 0 ? 0 : dcg / idcg
}

from math import log2
 
 
def is_hit(t: dict) -> bool:
    """Recall: single hits on any document, multi requires all of them"""
    if t["type"] == "none":
        return False
    if t["type"] == "multi":
        return all(doc_id in t["context_doc_ids"] for doc_id in t["answer_doc_ids"])
    return any(doc_id in t["context_doc_ids"] for doc_id in t["answer_doc_ids"])
 
 
def reciprocal_rank(t: dict) -> float:
    """The reciprocal of the first relevant result's rank. enumerate starts at 1, sparing an off-by-one"""
    for rank, doc_id in enumerate(t["ranked_doc_ids"], start=1):
        if doc_id in t["answer_doc_ids"]:
            return 1 / rank
    return 0.0
 
 
def ndcg_at(t: dict, k: int = 10) -> float:
    """Each of the top k discounted by 1 / log2(rank + 1), divided by the ideal ordering's score"""
    gains = [1 if d in t["answer_doc_ids"] else 0 for d in t["ranked_doc_ids"][:k]]
    dcg = sum(g / log2(i + 2) for i, g in enumerate(gains))
    idcg = sum(1 / log2(i + 2) for i in range(min(t["relevant_chunk_count"], k)))
    return dcg / idcg if idcg else 0.0

The unanswerable class enters none of those three and has its own: the refusal rate — of four unanswerable questions, how many the system genuinely said it could not find in the material for. The lab's baseline comes out like this:

Recall (answerable)   87.5%
Recall (single)       100.0%
Recall (multi)        50.0%
Mean reciprocal rank  1.0000
nDCG@10               0.6711
Refusal rate          0.0%

That set of numbers is informative. Single-document questions all hit while multi-hop hits only half — the "who approves a failover and what is their name" question never fetched the second document once. And the refusal rate is 0%: not one of four unanswerable questions stayed quiet, and all four were stuffed with irrelevant material. Counting recall alone, 87.5% looks quite respectable.

Two generation-side metrics: did it invent, did it answer

Retrieval finding the right thing still leaves the model able to twist it. Two generation-side metrics, and they must be read as a pair.

Faithfulness: for every statement in the answer, is there support in the material? It catches fabrication. The algorithm splits the answer one statement per line, judges each, and takes supported over total.

Answer relevancy: did the answer address the question? It catches saying a great deal without answering.

Why they must pair: "the material contains nothing that supports an answer" scores full faithfulness — it invented not one word — with zero relevancy. Watch faithfulness alone and your system learns that always refusing scores full marks. Conversely, watching relevancy alone, a well-argued and entirely on-topic falsehood also scores highly. These two metrics watch each other.

The lab's baseline has one pair particularly worth reading: faithfulness 1.0000 and answer relevancy 0.3931. Faithfulness is perfect because offline mode's "answer" is sentences lifted from hit chunks, verbatim from the material and therefore never invented; and relevancy at 0.39 says what was lifted is mostly not the sentence the question wanted. High faithfulness does not mean a correct answer, and that is this section's most memorable line.

Per-statement judging looks like this:

export function faithfulness(answer, contextText) {
  const unsupported = []
  let decidable = 0
  for (const claim of claimsOf(answer)) {
    const ratio = overlapRatio(claim, contextText)
    if (ratio === null) continue // the undecidable does not enter the denominator; see the pothole below
    decidable += 1
    if (ratio < MIN_OVERLAP) unsupported.push(claim)
  }
  return { score: decidable === 0 ? 0 : 1 - unsupported.length / decidable, unsupported }
}

def faithfulness(answer: str, context_text: str) -> tuple[float, list[str]]:
    unsupported: list[str] = []
    decidable = 0
    for claim in claims_of(answer):
        ratio = overlap_ratio(claim, context_text)
        if ratio is None:  # the undecidable does not enter the denominator
            continue
        decidable += 1
        if ratio < MIN_OVERLAP:
            unsupported.append(claim)
    score = 0.0 if decidable == 0 else 1 - len(unsupported) / decidable
    return score, unsupported

The pitfalls of a model as judge

A judgment like faithfulness ultimately needs a model — send the material, the question, and the answer together and have it score. That is LLM-as-judge. It is cheap, fast, and scalable, and it has three systematic biases you must know.

Position preference. Asked to choose between two answers, it leans toward the one presented first. The mitigation is not doing pairwise comparison and instead scoring each independently; when comparison is unavoidable, swap the two answers' order and run again, calling any disagreement a draw.

A preference for long answers. Given the same information, the longer write-up scores higher more easily. The mitigation is the previous section's per-statement judging: faithfulness is supported over total, so a longer answer has a larger denominator and every extra sentence must earn its own support, removing the length bonus automatically.

Judging itself. Using one model to both generate and judge makes it more lenient toward its own output. The mitigation is switching vendor or tier: generate with claude-sonnet-5 and judge with another vendor's model; on a tight budget claude-haiku-4-5-20251001 as judge is fine too, and after switching a manual spot check must be redone.

Beyond those three the prompt itself has two requirements: pin down the scoring anchors (what 1.0 means, what 0.6 means, what 0.3 means, since without anchors it scores differently every day), and require structured output listing the unsupported sentences verbatim. The second is for the manual spot check — given only 0.83 you cannot review anything, and with the disputed sentences listed you can judge in thirty seconds whether the judge was right.

And the only way to calibrate is a manual spot check computing agreement:

1. Run an evaluation, open the report, and sample 10 in a stratified way — every type represented, hits and misses both, and both high and low judge scores.
2. Answer one question per item: does this answer contain anything not written in the material? Judge faithfulness only, not quality.
3. Write your judgments into human-labels.json, compare against the judge's verdicts, and compute agreement.
4. Below 0.8, stop using that judge's scores to block a merge, go improve the prompt, and resample after improving it.

Wiring evaluation into the process

Writing the evaluation script is only step one; it has to become one command, one file, and one gate, or nobody will run it again three weeks later.

One command: pnpm start runs the whole suite and outputs reports/latest.json plus a human-readable Markdown. One file: save the version you accept as eval/baseline-report.json, the anchor for every later comparison. One gate:

// Watch five, each guarding a layer: recall guards finding it, reciprocal rank and nDCG guard
// ordering it, faithfulness guards not inventing, and the refusal rate guards staying quiet.
// Drop one and a class of regression slips through
export function gate(current, baseline, tolerance = 0.01) {
  const regressions = METRICS.map(({ name, pick }) => ({
    name,
    delta: Number((pick(current) - pick(baseline)).toFixed(4)),
  })).filter((r) => r.delta < -tolerance)
  return { passed: regressions.length === 0, regressions }
}

def gate(current: dict, baseline: dict, tolerance: float = 0.01) -> tuple[bool, list[dict]]:
    """Watch five, each guarding a layer: finding it, ordering it, not inventing, staying quiet"""
    regressions = [
        {"name": name, "delta": round(current[key] - baseline[key], 4)}
        for name, key in METRICS
        if current[key] - baseline[key] < -tolerance
    ]
    return not regressions, regressions

Run it in continuous integration and let a regression fail the pipeline:

MOCK=1 GATE=1 pnpm start   # exits non-zero when any metric regresses

Do not set the tolerance large. The same data with the same configuration should be bit-identical, and 0.01 exists only to allow for a change of embedding backend; set it to 0.05 and a genuine regression strolls straight through.

Finally, cost. Twenty questions through a model judge costs a few cents, so run it freely; at 200 questions ten times a day, somebody should be watching the bill. The common practice has two tiers: every commit runs only the retrieval metrics (pure local computation, seconds, free), with the model judge reserved for pre-merge and once nightly. Evaluation itself reports three bills too: the metric, the latency, and the money.

Source Reading

Hands-On Lab

starter/ has 5 exercise points cut out: the three retrieval metrics, faithfulness's per-statement judging, and the regression gate. Run as-is, 5 acceptance items are ❌ and each one you finish turns one ✅ — today's progress bar. The corpus and the 20-question golden set are prepared, and eval/golden-set.json's first 10 questions are verbatim D4's, so do not change one word; changing them makes the two days' numbers incomparable.

1. Open eval/golden-set.json, read all 20 questions, spot-check three against corpus/ to confirm the answer documents genuinely answer them, and confirm the four unanswerable questions genuinely have no answer in the corpus.
2. Implement the three retrieval metrics in src/metrics.ts, run once, watch recall go from 0.0% to a meaningful number, and note the gap between single and multi.
3. Add faithfulness's per-statement judging in src/judge.ts, run once, and watch section four's judge probe go from ❌ to ✅.
4. Follow the README to sample 10 in a stratified way, rewrite eval/human-labels.json, rerun, and read section five's agreement and disagreement list.
5. Wire the regression gate into src/report.ts, run once to confirm the gate blocked, then use MOCK=1 GATE=1 pnpm start to feel the non-zero exit code on a regression.

Interview Questions

Today's 4 questions are in the question bank below, weighted toward constructing an evaluation set, choosing retrieval metrics, and a model judge's reliability and biases. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Construct a golden set from real corpus data, and explain why it must include unanswerable questions and multi-hop questions
• Implement recall, Mean Reciprocal Rank, and Normalized Discounted Cumulative Gain, and explain what each is sensitive to
• Design a model-judge prompt and verify it's trustworthy with manual spot checks, rather than trusting its scores unconditionally
• Say which failure makes each of recall, reciprocal rank, and nDCG fall first
• Explain why faithfulness and answer relevancy must be read as a pair, and how watching one alone gets you fooled
• All 5 acceptance criteria of the lab pass, with a baseline report for week one's system in hand
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D9) brings week two's first optimization: hybrid search and reranking — fusing two retrieval paths with reciprocal rank fusion, then reranking the top few dozen with a cross-encoder. The reason for evaluating before optimizing: hybrid search's benefit depends entirely on the corpus and the question distribution, gaining a dozen points in some settings and not one point or even regressing in others. Without today's baseline, all you could say tomorrow is that it feels a bit better.