Why Retrieve at All: Hallucination, Knowledge Cutoffs, and the Cost of Long Context; a Minimal Keyword-Only RAG

Today's Goals

1. State in one sentence that Retrieval-Augmented Generation solves knowledge freshness, factual attribution, and context cost, and give one scenario where it shouldn't be used
2. Draw the five stages of Retrieval-Augmented Generation — chunking, indexing, retrieval, context assembly, generation — and explain how each stage can fail
3. Hand-write a BM25 scoring function and explain what term-frequency saturation and document-length normalization each guard against

Today installs no vector store, calls no embedding endpoint, and pulls in no framework: about three hundred lines of code get a complete question-answering path running. Once you have read this and finished the lab, scroll back to the top and tick off the three goals.

Plain-Language Walkthrough

Why an open-book exam beats memorizing, and costs less

Picture two exams. The first is closed-book: you memorize the whole textbook and answer from memory. However well you memorized, three problems remain — the textbook was revised last year and you did not know; where your memory is fuzzy you instinctively smooth out an answer that looks plausible; and asked by the marker which page supports that, you cannot say. The second is open-book: you memorize nothing and practice one thing only — hearing the question, quickly judging which page to turn to, reading that page properly, writing the answer from it, and marking the page number after it.

This course is about doing well in the second kind of exam from beginning to end. Its formal name is Retrieval-Augmented Generation (RAG): when a user asks, first fetch the most relevant passages from an external library of material, hand them to the model with the question, and have the model answer strictly from those passages. The model is still the same model; what changed is what it has at hand while answering.

Where does open-book win? In three very concrete things. Change one line in the library and the next question reflects it immediately, with nothing retrained; an answer can be traced to a specific passage, so a problem can be investigated; and each question carries only the relevant passages to the desk rather than the entire library, which differs by an order of magnitude on the token bill. Those three are knowledge freshness, factual attribution, and context cost — the first judgment to build today, and every day of the next thirteen is about making those three solid.

This analogy runs through the whole course, so map the words now: the corpus is the collection, chunking is cutting whole books into entries, the index is the card catalog, retrieval is fetching a book from the shelf by the catalog, reranking is the librarian picking again from the armful you fetched, a citation is a reference at the end of a paper, and refusal is saying "I could not find it" after searching the catalog rather than inventing an entry.

The model's three ailments are exactly three bills

Break the vague claim that a model is not enough into three verifiable phenomena.

The first ailment is the knowledge cutoff. A model's knowledge comes from its training data and it knows nothing that happened after training ended. Ask about the expense policy your company revised last week and it can only answer from the old version, in the same confident tone it uses for everything else. That is not the model being dim; it simply has no channel through which to know.

The second ailment is hallucination. A model works by predicting the most likely next word, and most likely and true are two different things. When it is unsure, what comes out is not "I do not know" but a passage that looks very much like a correct answer — right format, right terminology, confident tone, and invented content. That is far more dangerous than the first: a knowledge cutoff is a definite boundary and hallucination has none.

The third ailment is the absence of attribution. Even when it answers correctly this time, you do not know on what basis, and cannot point the answer back to source material you could check. For personal use that is merely inconvenient; for an enterprise knowledge base, support, or legal work, where the evidence is often matters more than whether the answer is right — a correct answer with no evidence cannot be used in a process.

Those three ailments correspond to three bills: the bill for updating knowledge, the bill for errors, and the bill for tokens. Every technical choice from today on gets settled against those three.

Three roads: retrieval, fine-tuning, and stuffing the whole book in

Making a model know something it originally did not has three engineering roads, each with its own cost curve.

The first is fine-tuning: continue training the model on your data, welding the knowledge into the parameters. It is good at teaching a model a style or a format — writing weekly reports in the company's voice, emitting fields in a fixed schema. It is bad at instilling facts: any change means retraining on a timescale of days, and even after training there is still no attribution.

The second is stuffing the whole book into the context: with windows now running to hundreds of thousands of tokens, a small knowledge base genuinely can go in whole. The upside is almost no engineering; the downside is paying for all the material on every single question, with latency rising with input length. Worse, as material grows, a model's reliability at locating key information in a long context declines — you will see it read without reading.

The third is retrieval: material lives outside and only the relevant passages are fetched each time. Changing material means changing a file, effective immediately; cost depends only on the passages fetched rather than the library's total size; and every passage carries an id so the answer can cite. The price is building a retrieval system yourself, and that system will itself make mistakes — the next thirteen days are mostly about that.

Choosing is not one of three but a question of what you lack. Lacking how to say something, fine-tune; lacking what to know, retrieve; and the two are frequently used together. As for when not to retrieve: if your task does not depend on external facts at all — rewriting a passage more concisely, converting code from one style to another — forcing retrieval in only adds noise, latency, and cost. The criterion is one sentence: would this task's answer change if a document changed? If not, you do not need retrieval.

Five stages, five ways to break

A retrieval-augmented generation system has only five stages, strung left to right:

Source documents → 1 chunking → 2 indexing → 3 retrieval → 4 context assembly → 5 generation → a cited answer

The beginner's most common error is treating that chain as a black box and, when something goes wrong, saying only that the model answered badly. In fact each stage has its own typical failure with very different symptoms:

• Chunking broke: a complete rule was cut in half, the first clause in one chunk and the second in the next. Retrieval hit one of them, and that one alone is incomplete or even misleading. The symptom is a half-right answer.
• Indexing broke: parsing was not clean, table rows crossed, headers and footers mixed into the body. The symptom is obviously irrelevant material in the retrieval results.
• Retrieval broke: the correct answer never entered the candidates. This is the most fatal, because no later stage can help — what is not in the material, the model can only invent. The symptom is an off-topic answer or confident nonsense.
• Context assembly broke: the material was right but the prompt's ordering got scrambled, it got truncated, or "answer only from the material below" was left out. The symptom is the model ignoring the material and answering from prior knowledge.
• Generation broke: material complete, instructions clear, and the model still missed a line or attributed one passage's number to another. The symptom is citation ids that do not match their content.

The way to use that table: investigate right to left, fix left to right. Right to left because the generated result is what you see first; left to right because errors on the left get amplified on the right, and no prompt rescues what retrieval failed to fetch. Day 8 turns this into quantifiable metrics; today, build the habit of attributing by stage.

Getting keyword retrieval solid: BM25 term by term

Now build stage 3. No vectors at all today — instead an algorithm settled in the 1990s and still standard in every retrieval system: BM25.

Start from the plainest idea: the more often a question's words appear in a document, the more likely it is relevant. That is term frequency. But frequency alone has a large hole — words like "the," "we," and "company" are everywhere and their counts say nothing about relevance. So each term needs a weight measuring how rare it is, which is inverse document frequency: a word appearing in only 2 documents is highly informative when it hits, and one appearing in 28 says essentially nothing.

Tokenize first. Chinese has no spaces and cannot be split on whitespace as English can. In a dependency-free setting the most economical approach is bigrams: cut a CJK run into overlapping two-character tokens, cut the same word in the query the same way, and they match without a dictionary. The price is roughly one token per character and an index twice the size — entirely irrelevant for thirty documents.

// CJK runs are cut into bigrams; letters and digits are cut into words; punctuation and whitespace separate
const SEGMENT = /[\u4e00-\u9fff]+|[a-z0-9]+/gu
const CJK = /^[\u4e00-\u9fff]+$/u
 
export function tokenize(text) {
  const tokens = []
  const segments = text.toLowerCase().match(SEGMENT) ?? []
  for (const seg of segments) {
    if (CJK.test(seg)) {
      if (seg.length === 1) {
        tokens.push(seg)
        continue
      }
      for (let i = 0; i + 1 < seg.length; i += 1) tokens.push(seg.slice(i, i + 2))
    } else if (seg.length > 1) {
      // A single letter or digit has almost no discriminating power; dropping them saves a lot of index
      tokens.push(seg)
    }
  }
  return tokens
}

import re
 
SEGMENT = re.compile("[\\u4e00-\\u9fff]+|[a-z0-9]+")
CJK = re.compile("^[\\u4e00-\\u9fff]+$")
 
 
def tokenize(text: str) -> list[str]:
    """CJK runs are cut into bigrams; letters and digits into words; punctuation and whitespace separate"""
    tokens: list[str] = []
    for seg in SEGMENT.findall(text.lower()):
        if CJK.match(seg):
            if len(seg) == 1:
                tokens.append(seg)
                continue
            tokens.extend(seg[i : i + 2] for i in range(len(seg) - 1))
        elif len(seg) > 1:
            # A single letter or digit has almost no discriminating power; dropping them saves index
            tokens.append(seg)
    return tokens

With tokens comes the inverted index: not which words a document contains but the reverse — which documents each word appears in, and how often. Retrieval then only has to score the candidate documents for the query's few terms rather than scanning the whole library. That is the value of a card catalog.

Then BM25 itself. The full formula looks like this:

score(D, Q) = Σ  idf(t) · ( f(t,D) · (k1 + 1) ) / ( f(t,D) + k1 · (1 - b + b · |D| / avgdl) )
             t∈Q
 
idf(t) = ln( 1 + (N - n(t) + 0.5) / (n(t) + 0.5) )

Intimidating, and it is only three parts.

The first part is the inverse document frequency, idf. N is the total document count and n(t) is how many contain this term. The rarer the term, the larger the numerator and the higher the weight. The ln around it compresses the magnitude, and the added 1 keeps the score from going negative when n(t) exceeds half the documents. This term brings stop-word behavior for free: with no stop-word list, a word like "the" pushes itself to near zero.

The second part is term-frequency saturation, controlled by k1. Note that f appears in both numerator and denominator, so the fraction approaches an upper bound as f grows rather than growing without limit. What does it guard against? Against an article writing "recycle bin" fifty times to dominate the ranking. Fifty times really is more relevant than five, and it should never be ten times as relevant. A smaller k1 saturates faster; at 0.2, appearing twice and appearing twenty times score almost the same.

The third part is length normalization, controlled by b. Long documents hit keywords by chance more easily — treat a whole book as one document and it can match almost any query. This term divides the document's length by the average to scale over-long documents back proportionally. At b of 0, length is ignored entirely; at 1, it is fully penalized; and 0.75 is the compromise long practice settled on and the default in nearly every implementation.

// idf: the fewer documents a term appears in, the more persuasive a hit on it is
export function idf(index, term) {
  const n = index.postings.get(term)?.size ?? 0
  return Math.log(1 + (index.docCount - n + 0.5) / (n + 0.5))
}
 
export function scoreDoc(index, docId, queryTerms, { k1 = 1.2, b = 0.75 } = {}) {
  const len = index.lengths.get(docId) ?? 0
  const norm = 1 - b + (b * len) / index.avgLength // the length term does not depend on the word; compute it once
  let score = 0
  for (const term of queryTerms) {
    const f = index.postings.get(term)?.get(docId) ?? 0
    if (f === 0) continue
    score += (idf(index, term) * (f * (k1 + 1))) / (f + k1 * norm)
  }
  return score
}

import math
 
 
def idf(index, term: str) -> float:
    """The fewer documents a term appears in, the more persuasive a hit on it is"""
    n = len(index.postings.get(term, {}))
    return math.log(1 + (index.doc_count - n + 0.5) / (n + 0.5))
 
 
def score_doc(index, doc_id: str, query_terms: list[str], k1: float = 1.2, b: float = 0.75) -> float:
    length = index.lengths.get(doc_id, 0)
    norm = 1 - b + b * length / index.avg_length  # the length term does not depend on the word
    score = 0.0
    for term in query_terms:
        f = index.postings.get(term, {}).get(doc_id, 0)
        if f == 0:
            continue
        score += idf(index, term) * (f * (k1 + 1)) / (f + k1 * norm)
    return score

The last stage is context assembly. Having fetched the top three passages, three things must happen while composing the prompt, and omitting any one produces the corresponding failure in today's lab: give each passage a citable id, write the citation rule into the instruction, and explicitly permit refusal. The third is the most often forgotten and is precisely the main gate against hallucination.

export function buildPrompt(question, passages) {
  const system = [
    'You are the knowledge base assistant for Skyladder Technologies. Answer only from the material below.',
    'When the material contains no answer, reply with exactly: The material contains nothing that supports an answer.',
    'Mark every conclusion with the source it came from in square brackets, in the form [doc-006].',
    'If two passages contradict each other, list both claims with their update dates rather than picking one.',
  ].join('\n')
 
  const context = passages
    .map((p) => `[${p.id}] (updated ${p.updated}) ${p.title}\n${p.body}`)
    .join('\n\n---\n\n')
 
  return { system, user: `Material:\n\n${context || '(none)'}\n\nQuestion: ${question}` }
}

def build_prompt(question: str, passages: list[dict]) -> tuple[str, str]:
    system = "\n".join(
        [
            "You are the knowledge base assistant for Skyladder Technologies. Answer only from the material below.",
            "When the material contains no answer, reply with exactly: The material contains nothing "
            "that supports an answer.",
            "Mark every conclusion with the source it came from in square brackets, in the form [doc-006].",
            "If two passages contradict each other, list both claims with their update dates rather than "
            "picking one.",
        ]
    )
    context = "\n\n---\n\n".join(
        f"[{p['id']}] (updated {p['updated']}) {p['title']}\n{p['body']}" for p in passages
    )
    return system, f"Material:\n\n{context or '(none)'}\n\nQuestion: {question}"

Why the first version uses no vectors

You may be asking: the world talks about nothing but vector search, so why does day one write an algorithm from thirty years ago?

Three reasons. First, it is explainable. Every point of a BM25 score decomposes into which term contributed how much, so when the ranking is wrong you can see on the spot whether it is a frequency problem or a rarity problem. Vector search hands you a similarity of 0.83 and no idea where to start when it goes wrong. Second, zero dependencies: no embedding endpoint, no vector store, no GPU, running in two minutes and reproducible on a laptop with no network. Third, and most importantly — it is the baseline.

A baseline is not the version you make do with; a baseline is a ruler. Over the next thirteen days you will add vectors, hybrid search, reranking, and query rewriting, each of which costs money, adds latency, and adds a failure point. What justifies any of it? Only one answer: against the baseline, how much did the metric rise, how much did latency rise, and how much did the money rise. Three bills, and missing one means it does not count.

BM25 also does something vectors cannot: exact matching. Document ids, error codes, personal names, model numbers — vectors often "understand" these into something semantically nearby, while BM25 goes by the letter. That is why on day 9, when we cover hybrid search, the keyword path is not dropped but run alongside the vector path and fused — what you write today lives all the way to the last day.

Today's lab shows the baseline's two real shortcomings: when an answer is split across two documents, BM25 fetches only one of them (the multi-hop problem, handled on day 12); and when two documents give different conclusions about the same thing, it takes both without knowing which to believe (the conflicting material problem, handled on day 6). Seeing the shortcomings is today's takeaway, because each of the following days patches one of them.

Source Reading

Hands-On Lab

The lab directory already holds a fictional corpus of thirty documents — the product manual, internal policies, technical documentation, support Q&A, and meeting notes of a company called Skyladder Technologies. The same corpus is used for all thirteen remaining days, so it is worth five minutes of browsing, especially doc-005 with its mangled table and doc-024 whose support answer disagrees with the product manual; they were left there deliberately.

starter/ has four exercise points cut out: tokenization, inverse document frequency, BM25 scoring, and score threshold filtering. Run as-is, every question shows not one term matched — because the default tokenizer splits on whitespace, which is useless for Chinese, and that is your first task. No API key is needed to finish it all offline; offline mode replaces only the final generation with template assembly, and the first four stages genuinely run.

1. Read three to five documents in corpus/ first and make sure you know what the corpus does and does not contain, since that determines what questions you can ask.
2. Complete the tokenizer and the inverted index, run again, and watch retrieval go from not one term matched to a scored top three.
3. Switch scoring to full BM25, compose the top three into the prompt, and complete one question-answer round, seeing an id such as [doc-006] in the answer.
4. Ask b08 and b09, two questions with no answer in the corpus, and watch one blocked by the score threshold while the other invents regardless; work out which layer is responsible for each.
5. Adjust K1 and B once each and rerun the baseline, comparing the changes in the top three, then keep the default-parameter baseline.json as the reference for the next thirteen days.

Interview Questions

Today's 4 questions are in the question bank below, weighted toward retrieval-augmented generation versus fine-tuning and the limits of long context, the meaning of the BM25 formula, and the failure modes of each stage in the minimal system. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• State in one sentence that Retrieval-Augmented Generation solves knowledge freshness, factual attribution, and context cost, and give one scenario where it shouldn't be used
• Draw the five stages of Retrieval-Augmented Generation — chunking, indexing, retrieval, context assembly, generation — and explain how each stage can fail
• Hand-write a BM25 scoring function and explain what term-frequency saturation and document-length normalization each guard against
• Explain why investigation runs right to left and fixes run left to right, and use it on one real wrong answer
• All 5 acceptance criteria of the lab pass, with baseline.json generated and kept
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D2) we give the system its other eye: embeddings and vector search. Today's BM25 goes by the letter, so rephrasing the same question leaves it stumped; vectors turn similar in meaning into near in coordinates, which covers exactly that gap. Learning keywords before vectors is deliberate — start with vectors and you come to believe retrieval simply is computing similarity, never again remembering that exact matching exists; whereas with a baseline in hand, every improvement tomorrow can be measured on the spot.