The Generation Side: Ordering Context, Labeling Citations, When You Must Refuse to Answer, and Streaming Responses

Laying material on the table: ordering, deduplication, budget, and that middle ground

A clerk laying exhibits on the table does it with care. So does a pile of retrieved chunks, passing four steps between a raw hit and the prompt.

Step one is ordering. By retrieval score, high to low, with no suspense. But equal scores need a tie-breaker: without a stable order, chunk numbers drift between runs and "cited chunk 3" in the log no longer matches anything. Break ties by chunk id — pure engineering discipline.

Step two is deduplication. The same passage often appears once in the product manual and once in the support Q&A. Duplicate material wastes budget and makes the model believe this point was stressed repeatedly and must be important. Fingerprint the normalized text and dedupe.

Step three is the budget. Give the material an explicit token budget (1,200 in the lab) and fill from the highest score down. The detail: when something does not fit, do not stop — a smaller chunk further down may still fit; and of course do not try forever, so add a maximum chunk count as a backstop.

Step four is the most counter-intuitive: reordering. A chunk's position in the context affects whether the model reads it, with the beginning and the end noticeably more likely to be used and the exact middle likeliest to be missed. The remedy is putting the most important at both ends: first place at the start, second at the end, third in position two, working inward. That way the closer to the middle a chunk sits the less important it already was, and the cost of missing it is smallest.

// The number is the presentation index: the model may cite only numbers issued here, and verification looks them back up
export function assemble(scored, { budget = 1200, maxBlocks = 5 } = {}) {
  // Break ties by chunkId, or chunk numbers drift between runs and the ids in the log stop matching
  const ranked = [...scored].sort(
    (a, b) => b.score - a.score || a.chunk.chunkId.localeCompare(b.chunk.chunkId)
  )
 
  const seen = new Set()
  const picked = []
  let usedTokens = 0
  for (const item of ranked) {
    if (picked.length >= maxBlocks) continue
    const key = fingerprint(item.chunk.text)
    if (seen.has(key)) continue // the manual and the support Q&A often carry the same passage
    const cost = countTokens(item.chunk.text)
    if (usedTokens + cost > budget) continue // not break: a smaller chunk further down may still fit
    seen.add(key)
    usedTokens += cost
    picked.push(item)
  }
 
  // The most important at both ends, the less important working inward
  const head = []
  const tail = []
  picked.forEach((item, i) => (i % 2 === 0 ? head.push(item) : tail.unshift(item)))
  const ordered = [...head, ...tail]
 
  return { blocks: ordered.map((item, i) => ({ n: i + 1, ...item })), usedTokens }
}

def assemble(scored: list[Scored], budget: int = 1200, max_blocks: int = 5) -> AssembleResult:
    """The number is the presentation index: the model may cite only numbers issued here"""
    # Break ties by chunk_id, or chunk numbers drift between runs and the ids in the log stop matching
    ranked = sorted(scored, key=lambda s: (-s.score, s.chunk.chunk_id))
 
    seen: set[str] = set()
    picked: list[Scored] = []
    used_tokens = 0
    for item in ranked:
        if len(picked) >= max_blocks:
            continue
        key = fingerprint(item.chunk.text)
        if key in seen:  # the manual and the support Q&A often carry the same passage
            continue
        cost = count_tokens(item.chunk.text)
        if used_tokens + cost > budget:  # not break: a smaller chunk further down may still fit
            continue
        seen.add(key)
        used_tokens += cost
        picked.append(item)
 
    # The most important at both ends, the less important working inward
    head = picked[0::2]
    tail = picked[1::2][::-1]
    ordered = head + tail
 
    blocks = [Block(n=i + 1, chunk=it.chunk, score=it.score) for i, it in enumerate(ordered)]
    return AssembleResult(blocks=blocks, used_tokens=used_tokens)

Labeling citations so they can be verified

Now to today's core. Having the model attribute its sources has two forms, and they differ greatly.

The first is free text: it writes "according to Drive and File Management." Such a citation cannot be verified — a title is a string the model can produce offhand, and you would need fuzzy matching to know which document is meant, unable to tell when matching fails whether it invented the title or your matching is simply weak.

The second is a chunk number: assembly gives each piece of material a number, the prompt states plainly that only the numbers sent may be cited, and the model outputs "the recycle bin keeps items 30 days [2]". A number is a closed set — you issued 1 through 5, so 8 is certainly invented, a judgment needing no fuzzy matching and one line of code. Verifiability comes from a closed set, not from phrasing.

The output format has one more choice. Non-streaming is most robust as JSON: one conclusion with an array of numbers, validated once against a schema. Streaming cannot use JSON — it needs the closing brace to parse, so there is nothing to stream — and falls back to plain text with inline markers. Two contracts, and verification must be the same one.

Then the clerk's job, in two gates:

Gate one: the number must exist. The most common shape of a fabricated citation is a number slightly above the largest one you issued. That is easy to catch.

Gate two: the cited chunk must substantively overlap this sentence. That gate is the crucial one, because the subtler fabrication has a real number and false content — it cites chunk 2 while chunk 2 discusses database sharding, and an existence check waves it through. The criterion is an overlap ratio: tokenize the sentence and see what fraction can be found in the cited chunk's original text, failing below a threshold.

One detail that must be handled: remove tokens appearing in most chunks before computing overlap. Words like "Skyladder" and "file" are everywhere, and computing overlap with them lets any cited chunk pass. That is the same reasoning as BM25 suppressing common words with inverse document frequency (the formula is in D1).

// Substantive overlap: what fraction of this sentence's tokens can be found in the cited chunk
export function overlapRatio(sentence, blockText, common) {
  // Deduplicate and remove tokens appearing in most chunks — otherwise any citation passes
  const terms = [...new Set(tokenize(sentence))].filter((t) => !common.has(t))
  if (terms.length === 0) return 0
  const inBlock = new Set(tokenize(blockText))
  return terms.filter((t) => inBlock.has(t)).length / terms.length
}
 
export function verifyAnswer(claims, blocks, minOverlap = 0.3) {
  const byNumber = new Map(blocks.map((b) => [b.n, b]))
  const common = commonTerms(blocks)
 
  return claims.map((claim) => {
    const issues = []
    const verified = []
    for (const n of claim.citations) {
      const block = byNumber.get(n)
      if (!block) {
        issues.push(`cited a nonexistent chunk number [${n}]`) // gate one: the numbers are a closed set
        continue
      }
      // Note the comparison uses the original chunk.text, not the compressed version put in the prompt
      const ratio = overlapRatio(claim.text, block.chunk.text, common)
      if (ratio < minOverlap) {
        issues.push(`[${n}] has only ${ratio.toFixed(2)} substantive overlap with this sentence`) // gate two
        continue
      }
      verified.push(n)
    }
    return { claim, ok: issues.length === 0, issues, verified }
  })
}

def overlap_ratio(sentence: str, block_text: str, common: set[str]) -> float:
    """Substantive overlap: what fraction of this sentence's tokens can be found in the cited chunk"""
    # Deduplicate and remove tokens appearing in most chunks — otherwise any citation passes
    terms = {t for t in tokenize(sentence) if t not in common}
    if not terms:
        return 0.0
    in_block = set(tokenize(block_text))
    return len(terms & in_block) / len(terms)
 
 
def verify_answer(claims: list[Claim], blocks: list[Block], min_overlap: float = 0.3):
    by_number = {b.n: b for b in blocks}
    common = common_terms(blocks)
 
    verdicts = []
    for claim in claims:
        issues: list[str] = []
        verified: list[int] = []
        for n in claim.citations:
            block = by_number.get(n)
            if block is None:
                issues.append(f"cited a nonexistent chunk number [{n}]")  # gate one
                continue
            # Note the comparison uses the original chunk.text, not the compressed prompt version
            ratio = overlap_ratio(claim.text, block.chunk.text, common)
            if ratio < min_overlap:
                issues.append(f"[{n}] has only {ratio:.2f} substantive overlap with this sentence")
                continue
            verified.append(n)
        verdicts.append(Verdict(claim=claim, ok=not issues, issues=issues, verified=verified))
    return verdicts

What happens when verification fails? Write the specific reason as feedback and send it back with the original material for one regeneration. The feedback must be specific to the sentence: "this sentence cited a nonexistent chunk number 8." The second version is usually well behaved, but give only one chance — two fabricating versions in a row mean the material does not support it, and the next section's refusal applies.

One last thing: citations must be clickable. Passing verification is an internal conclusion, and only a user able to click the number and see that passage makes it genuinely verifiable, so the server needs an endpoint fetching original text by chunk id. Without it the evidence chain is broken at the user's end.

Refusal is not failure: three lines, three phrasings

"Could not find it" is not a fault in knowledge base question answering but correct output. The actual fault is inventing a passage after failing to find one. The three lines differ in trigger timing, criterion, and phrasing, and mashing them into one "sorry, I do not know" wastes the work.

Line one: retrieval scores too low. The top score below a threshold means the store holds no relevant material. This is judgeable before generation, and refusing early saves a whole model call. The phrasing should give the user a next action: rephrase and ask again, or confirm whether the material has been ingested. How is the threshold set? Set too high, answerable questions get blocked (a user seeing "could not find it" while the material is in the store is the most trust-damaging error); set too low, noise still enters the context. The only reliable method is running a batch of known-answerable and known-unanswerable questions and reading the score distribution, not guessing. Once D8 builds evaluation, it becomes a tunable number.

More importantly, this line cannot cover every case that should refuse. D1's baseline holds a ready-made counter-example pair: b08 (the team outing budget) has no answer in the corpus and scores around 3, so the threshold blocks it; b09 (expensing a phone bill) likewise has no answer, and a page full of "expense" pushes the score to 9.41 where the threshold cannot block it. (Both numbers come from D1's document-level baseline; today's retrieval unit is the chunk and the absolute scores differ.) D1 could only cover b09 with a refusal instruction in the prompt, and a prompt cannot cover it — today moves that gate out of the prompt and into code, which is line three.

Line two: materials contradict each other. Retrieval fetched both the 200 MB and the 100 MB documents, and the model's default is to pick one — and you cannot even see it picking. The right approach is not letting it pick at all: code first finds different numbers about the same thing across chunks. Three criteria must hold together: they come from different documents, the units match, the values differ, and the context before the two numbers shares at least two tokens — that last one is the key against false positives, since without it "the recycle bin keeps items 30 days" and "the trial period is 90 days" get judged a conflict.

Once detected, there are two ways to handle it, and which one is a product decision, not a technical one. One is presenting both: lay out both claims with their sources and update dates and hand the choice back to a person, which is what the lab does. The other is choosing one: take the newer by update date, but only when you have an additional authority signal — the corpus's doc-028 weekly meeting note points out this inconsistency and assigns an action item, and with backing like that choosing one holds up. Without backing, present both honestly. Picking for the user and not telling them another claim exists is this section's easiest mistake.

// Covers only conflicts of the numbers-with-units kind. Textual contradictions need a model, which costs more and misjudges more
export function detectConflict(blocks) {
  const all = blocks.map((b) => ({ block: b, list: extractQuantities(b.chunk.text) }))
  for (let i = 0; i < all.length; i += 1) {
    for (let j = i + 1; j < all.length; j += 1) {
      const [a, b] = [all[i], all[j]]
      if (a.block.chunk.docId === b.block.chunk.docId) continue // different numbers in one document are usually not a conflict
      for (const qa of a.list) {
        for (const qb of b.list) {
          if (qa.unit !== qb.unit || qa.value === qb.value) continue
          // Context keys: tokens in the fourteen characters before the number. Without this, 30 days and 90 days read as a conflict
          const shared = [...qa.keys].filter((k) => qb.keys.has(k))
          if (shared.length < 2) continue
          return { unit: qa.unit, left: describe(a.block, qa), right: describe(b.block, qb) }
        }
      }
    }
  }
  return null
}

def detect_conflict(blocks: list[Block]) -> ConflictEvidence | None:
    """Covers only conflicts of the numbers-with-units kind. Textual contradictions need a model"""
    all_q = [(b, extract_quantities(b.chunk.text)) for b in blocks]
    for i, (block_a, list_a) in enumerate(all_q):
        for block_b, list_b in all_q[i + 1 :]:
            if block_a.chunk.doc_id == block_b.chunk.doc_id:
                continue  # different numbers in one document are usually not a conflict
            for qa in list_a:
                for qb in list_b:
                    if qa.unit != qb.unit or qa.value == qb.value:
                        continue
                    # Context keys: tokens before the number. Without this, 30 days and 90 days read as a conflict
                    if len(qa.keys & qb.keys) < 2:
                        continue
                    return ConflictEvidence(
                        unit=qa.unit,
                        left=describe(block_a, qa),
                        right=describe(block_b, qb),
                    )
    return None

Line three: the question falls outside what the material covers. This can only be judged after generation, and it plugs exactly the b09 gap above: the material looks relevant and never mentions phone bills at all. The criterion follows naturally — after citation verification, not one valid citation remains, meaning no conclusion stands. The phrasing must differ clearly from line one: not "no relevant material found" but "relevant documents were found and none of them contains anything that directly answers this question," and those two give the user entirely different next actions.

Context compression: let a small model delete a pass first

There is one more optimization after material is on the table: before handing it to the expensive model, let a cheap small one delete the irrelevant sentences.

Retrieval returns whole chunks, and one chunk usually holds one or two sentences genuinely answering the question with the rest being neighbors from the same section. Saving money is incidental; the main gain is less noise and higher accuracy — the fewer irrelevant sentences, the smaller the chance the model attributes one passage's number to another. The lab uses the cheap claude-haiku-4-5-20251001 for this step with claude-sonnet-5 still generating.

Report all three bills together. Metric: a more focused answer, quantifiable only once D8 builds evaluation. Latency: one extra model call, and a small model is fast and still a real round trip, so compressing every chunk means several concurrent ones. Cost: saving the main model's input tokens and paying for one small model call — worthwhile with larger chunks and a more expensive main model, and a straight loss when the material was short anyway.

Self-test output (solution, MOCK=1):
✅ Context compression: irrelevant sentences deleted
   651 tokens before → 530 after, saving 18.6%

That 18.6% came from this thirty-document corpus and this question, and another corpus may well give a different number. Its significance is not the magnitude but that your system must have a place that can print that line — without it you have no idea whether compression saved anything or merely added a call.

Compression's most dangerous property is that the small model rewrites along the way: writing "preserve verbatim" in the prompt does not stop it, and it loves condensing two sentences into a smoother one. So the code needs a hard constraint: every retained sentence must be findable verbatim in the original, and failing that the whole chunk reverts to the uncompressed version. One equally important discipline: citation verification always compares against the original, never the compressed product. What the user opens is the original, and once the verification target is not what the user sees, "verification passed" guarantees nothing.

Streaming and citations are a contradiction

The final trade-off. Streaming's selling point is characters as early as possible, and citation verification requires a finished sentence before it can check — those two naturally conflict.

The server pushes the answer to the browser as a series of events over a text protocol such as SSE, and the messages look like this:

event: meta
data: {"blocks":[{"n":1,"chunkId":"doc-010#c02"}],"contextTokens":530}
 
event: sentence
data: {"text":"The open API's default rate limit is 600 requests per minute","citations":[{"n":1,"chunkId":"doc-010#c02"}]}
 
event: dropped
data: {"text":"An administrator can change this value to any size in the console","detail":["[2] has only 0.03 substantive overlap"]}
 
event: done
data: {"status":"answered","emitted":1,"dropped":1}

The plainest approach is sending whatever arrives. That road does not work, because emitted characters cannot be withdrawn: by the time you discover at the end that the third sentence's citation was invented, that sentence is already on the user's screen and all you can do is pop up "please ignore that last sentence" — worse than not streaming.

The compromise is buffering by sentence. Accumulate a full sentence, verify it, emit it with its verified citations if it passes, and drop the whole sentence if it does not. The cost is explicit: time to first output goes from one token to one sentence, typically two or three hundred milliseconds, which users barely notice; whereas a wrong citation reaching the screen costs trust.

let buffer = ''
for await (const delta of streamText(question, blocks)) {
  buffer += delta
  // Handle only complete sentences. Half a sentence cannot be verified, since the citation marker precedes the full stop
  const { sentences, rest } = takeSentences(buffer)
  buffer = rest
  for (const raw of sentences) {
    // Strip [1][3] out of the text: the prose stays clean and the numbers go to verification separately
    const claim = parseInlineSentence(raw)
    const verdict = verifyAnswer([claim], blocks)[0]
    if (claim.citations.length > 0 && verdict.verified.length === 0) {
      send('dropped', { text: claim.text, detail: verdict.issues }) // the whole sentence is not emitted
      continue
    }
    send('sentence', { text: claim.text, citations: citationsOf(blocks, verdict.verified) })
  }
}

buffer = ""
async for delta in stream_text(question, blocks):
    buffer += delta
    # Handle only complete sentences; half a sentence cannot be verified
    sentences, buffer = take_sentences(buffer)
    for raw in sentences:
        # Strip [1][3] out of the text: the prose stays clean and the numbers go to verification separately
        claim = parse_inline_sentence(raw)
        verdict = verify_answer([claim], blocks)[0]
        if claim.citations and not verdict.verified:
            await send("dropped", {"text": claim.text, "detail": verdict.issues})
            continue
        await send(
            "sentence",
            {"text": claim.text, "citations": citations_of(blocks, verdict.verified)},
        )

There is one more ordering nicety: a refusal must be sent before the stream begins. Too-low scores and contradicting materials are both judged before generation, so push a refusal event and close the stream, and the user never sees half an answer withdrawn. Line three cannot be judged that early, but under sentence buffering it manifests as not one sentence emitted, with a refusal event appended at the close. That is why the three lines were separated by before or after generation — the separation is not only code structure; it decides directly what the user sees while streaming.

Source Reading

Hands-On Lab

Confirm one thing before starting: today's retrieval uses D1's BM25 unchanged, and everything to build is after the material returns. The starter is a complete running service with four gaps, each left with a default behavior that makes the problem visible — citation verification checking only existence, for instance, so the real-number-false-content fabrication sails through.

1. Run the starter's self-test once, see 4 passing and 4 failing, and understand each failure's cause before touching anything.
2. Complete the assembly function's budget control and end-weighted reordering, and watch the chunk number order become top score at the start and second place at the end.
3. Complete substantive overlap and verify with a request carrying the fabrication switch: version 1 is rejected with two specific reasons and version 2 passes.
4. Complete conflict detection, ask about the upload limit once, and watch the answer present both numbers with their update dates rather than picking one for you.
5. Change the streaming route to buffer by sentence and observe the two fabricated sentences appearing as dropped events, with not one wrong number in the prose.

Interview Questions

Today's 4 questions are in the question bank below, weighted toward the effect of context assembly order, verifying citation credibility, setting the refusal threshold, and how not to let a wrong citation escape while streaming. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Design a prompt structure that makes the model answer only from the given material and cite sources by chunk number
• Implement post-hoc citation verification that catches sources the model fabricated before they reach the user
• Explain the two triggers for refusing to answer — too-low retrieval scores and contradicting materials — and write a handling strategy for each
• Explain why verifiability comes from a closed set rather than from phrasing, with a concrete reason free-text citations cannot be verified
• State each refusal line's criterion, trigger timing (before or after generation), and difference in phrasing
• All 5 acceptance criteria of the lab pass, with all 8 self-checks green
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D7) assembles six days of parts into a genuine service: one ingest command, one question-answering endpoint, one configuration document, brought up with container orchestration in a single command. Why that order? Because until today you hold six piles of individually runnable code, and between running and shippable lie three things: module boundaries, configuration, and startup order. Filling those in closes week one, and only then is it comfortable to set evaluation's ruler against it in week two.