RAG, Level Up: Hybrid Search, Reranking, Citations, Recall Evaluation

The two roads' blind spots do not overlap, and their scores cannot be added

D12 finished the vector road: chunking, embeddings, cosine similarity, wrapped into a memory_search tool. All of that carries over unrepeated — 1536 dimensions, text-embedding-3-small, 0.02 dollars per million tokens, the same set.

Today adds a keyword road beside it and answers one question: what exactly did it fill in?

Today's lab's knowledge base is 54 e-commerce support documents cut into 64 chunks. Fire one real query at it:

query: what does E4032 mean, and can I ask the buyer to retry
  vector top5   - s07#0 (the payment error-code document) is absent
  keyword top1  - s07#0

That is what a blind spot looks like: the vector road pushed the correct answer out of the top five, and the keyword road ranked it first. The reverse exists too: ask when postage can be waived, the documents say "shipping fee," the keyword road matches not one character, and only vectors bring it back.

State what the keyword road computes. BM25's intuition is two lines: the more a term appears in a document the more relevant it is; and the more common that term is across the corpus the less it is worth. So "refund" has almost no discriminating power in an e-commerce knowledge base, while E4032 appears in one document and hitting it is nearly certain identification. Vector search is least sensitive precisely to this kind of low-frequency, semantically empty string — compressed into 1536 dimensions it crowds in with every other identifier. The two methods' strengths are exactly complementary, which is no coincidence but a consequence of how each computes.

How is the keyword road built? The teaching implementation uses Postgres's full-text search (tsvector plus ts_rank) without pulling in a search engine. There is a trap that must be stated for languages without spaces: Postgres's default tokenizer effectively does not tokenize Chinese, so a whole sentence becomes one token and nothing is ever found. The teaching fallback splits into adjacent-character bigrams, while English words and identifiers (sf-3000, e4032) are kept whole and must not be shredded. Production needs a proper CJK tokenizer extension; do not treat bigrams as the destination.

// CJK splits into adjacent-character bigrams; English and identifiers stay whole -
// shred e4032 and it can never be found again
export function bigrams(text) {
  const lowered = text.toLowerCase()
  const tokens = []
  // Pull out runs of alphanumerics (including hyphens) as single tokens first
  for (const word of lowered.match(/[a-z0-9][a-z0-9-]*/g) ?? []) tokens.push(word)
  const han = lowered.replace(/[^\p{Script=Han}]/gu, ' ')
  for (const run of han.split(/\s+/).filter(Boolean)) {
    if (run.length === 1) tokens.push(run)
    for (let i = 0; i + 1 < run.length; i++) tokens.push(run.slice(i, i + 2))
  }
  return tokens
}

import re
 
_WORD = re.compile(r"[a-z0-9][a-z0-9-]*")
_NON_HAN = re.compile(r"[^\u4e00-\u9fff]")  # the CJK range, escaped
 
 
def bigrams(text: str) -> list[str]:
    lowered = text.lower()
    # Alphanumeric words stay whole, and CJK then splits into bigrams
    tokens = _WORD.findall(lowered)
    for run in _NON_HAN.sub(" ", lowered).split():
        if len(run) == 1:
            tokens.append(run)
        tokens.extend(run[i : i + 2] for i in range(len(run) - 1))
    return tokens

// Dependencies: JDK 17+ standard library
static final Pattern WORD = Pattern.compile("[a-z0-9][a-z0-9-]*");
 
static List<String> bigrams(String text) {
    var lowered = text.toLowerCase(Locale.ROOT);
    var tokens = new ArrayList<String>();
    WORD.matcher(lowered).results().forEach(m -> tokens.add(m.group()));
    // \p{IsHan} is a JDK character class, so no hand-written code-point ranges
    for (var run : lowered.replaceAll("[^\\p{IsHan}]", " ").split("\\s+")) {
        if (run.isEmpty()) continue;
        if (run.length() == 1) tokens.add(run);
        for (int i = 0; i + 1 < run.length(); i++) tokens.add(run.substring(i, i + 2));
    }
    return tokens;
}

import Foundation
 
func bigrams(_ text: String) -> [String] {
    let lowered = text.lowercased()
    var tokens: [String] = []
    // Swift's String is a collection of Characters, so slicing by Character follows
    // grapheme boundaries and never breaks a combining sequence
    for word in lowered.split(whereSeparator: { !$0.isLetter && !$0.isNumber && $0 != "-" })
    where word.allSatisfy({ $0.isASCII }) {
        tokens.append(String(word))
    }
    let han = lowered.map { $0.unicodeScalars.first.map(isHan) == true ? $0 : " " }
    for run in String(han).split(separator: " ") {
        let chars = Array(run)
        if chars.count == 1 { tokens.append(String(chars[0])) }
        for i in 0 ..< max(0, chars.count - 1) { tokens.append(String(chars[i ... i + 1])) }
    }
    return tokens
}
 
private func isHan(_ scalar: Unicode.Scalar) -> Bool {
    (0x4E00 ... 0x9FFF).contains(Int(scalar.value))
}

Now there are two candidate lists, so how do they combine into one?

The intuitive move is a weighted sum: 0.6 times cosine similarity plus 0.4 times BM25. That road does not work, and not because weights are hard to tune but because the two scores are simply not comparable: cosine similarity sits between 0 and 1 with a dense distribution, often separating candidates by 0.02; BM25 has no upper bound and a document hitting three rare terms can score 12. Add them and BM25 unilaterally decides the outcome — tune the weights all night and you are only tuning how much BM25 gets to dictate.

The correct move uses only ranks, never scores: RRF (Reciprocal Rank Fusion). Each road's rank converts to 1 / (k + rank), one document's scores across roads are summed, and k is fixed at 60.

Why is k 60? It is an empirical value fixed by experiment when the method was proposed, and its role is flattening the gaps between the top few: without it (k = 0) first place scores 1 and second 0.5, a steep drop that lets each road's winner dictate; at 60, first place is 1/61 and tenth 1/70, a gap of barely a tenth, so "top twenty on both roads" outweighs "first on one road." That is exactly the weighting we want — consensus over single-point confidence. Sixty is no magic number, and before you have your own evaluation set there is no reason to move it.

Why this works: a rank is dimensionless. First is first, whether the raw score was 0.83 or 12.6. So documents ranked highly by both roads float to the top, and a document only one road recognizes can still push in on that road's high rank — which is what we want, because the blind spots do not overlap.

The lab has an example you can compute by hand: two rankings of [a, b, c] and [c, d, a] fuse with a first, scoring 1/61 + 1/63 which is about 0.03227. The starter's "add the scores directly" version gives c first on the same input, scoring 12.635 — c is merely first on the second road, and its raw BM25 score crushes everything.

That example hides one detail that must be handled: a and c are tied (each is first on one road and third on the other, so 1/61 + 1/63 is exactly equal). If who comes first on a tie is left to a hash table's iteration order, the same input gives different results across languages and runs — and of today's four implementations, only those explicitly adding "break ties by id" agree everywhere. Retrieval results must be reproducible, or your evaluation set measures a different number every time.

export const RRF_K = 60
 
// The input is several ranked id lists (each already sorted by its own score).
// Scores are discarded entirely; only ranks are used.
export function rrfFuse(rankings) {
  const scores = new Map()
  for (const ranking of rankings) {
    ranking.forEach((id, index) => {
      // index starts at 0 and ranks start at 1
      scores.set(id, (scores.get(id) ?? 0) + 1 / (RRF_K + index + 1))
    })
  }
  // Break ties by id: without this the result depends on Map iteration order and is
  // not reproducible
  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0]))
    .map(([id]) => id)
}

RRF_K = 60
 
 
def rrf_fuse(rankings: list[list[str]]) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for index, doc_id in enumerate(ranking):
            # Ranks start at 1; scores are discarded
            scores[doc_id] = scores.get(doc_id, 0.0) + 1 / (RRF_K + index + 1)
    # Break ties by id ascending: sort by id first, then a stable sort by score descending
    return sorted(sorted(scores), key=scores.__getitem__, reverse=True)

// Dependencies: JDK 17+ standard library
static final int RRF_K = 60;
 
static List<String> rrfFuse(List<List<String>> rankings) {
    var scores = new HashMap<String, Double>();
    for (var ranking : rankings) {
        for (int index = 0; index < ranking.size(); index++) {
            // merge does one lookup fewer than getOrDefault plus put and cannot miss the initial value
            scores.merge(ranking.get(index), 1.0 / (RRF_K + index + 1), Double::sum);
        }
    }
    // A HashMap is unordered, so ties must be broken explicitly or a new JVM changes the result
    return scores.entrySet().stream()
        .sorted(Map.Entry.<String, Double>comparingByValue().reversed()
            .thenComparing(Map.Entry.comparingByKey()))
        .map(Map.Entry::getKey)
        .toList();
}

let rrfK = 60
 
func rrfFuse(_ rankings: [[String]]) -> [String] {
    var scores: [String: Double] = [:]
    for ranking in rankings {
        // enumerated()'s offset starts at 0, so add 1 for the rank
        for (index, id) in ranking.enumerated() {
            scores[id, default: 0] += 1 / Double(rrfK + index + 1)
        }
    }
    // A Dictionary is unordered and sorted(by:) is not stable, so ties need an explicit id tiebreak
    return scores.sorted { ($0.value, $1.key) > ($1.value, $0.key) }.map(\.key)
}

Reranking: expensive, so do it on few candidates

The fused list is still ordered by retrieval signals, not by how relevant each item is to the question. Reranking compares each candidate against the question in detail.

The teaching implementation uses batch LLM scoring: send 40 candidates and the question to a model at once, have it score each 0 to 10, and reorder. The upside is not pulling in another model; the downside is that it is slow, expensive, and its scores drift with the prompt's wording.

Production does it differently, and the difference in convention must be stated: production uses a purpose-trained cross-encoder reranker (question and candidate concatenated into one encoder, outputting relevance directly). The price is one extra 100-to-300-millisecond call plus a machine to run inference on — it is not an API call billed per token, and you have to provision resources for it. So cap the reranked candidate count: 40 is a teaching figure and production commonly caps at 50 to 100.

Do not present the teaching version as standard practice. Saying in an interview that you rerank by having an LLM score retrieval results invites the follow-up "how did you account for latency and cost, and why not a dedicated reranker" — and being unable to answer gives you away.

Citations: the model will cite a number you never gave it

Retrieved material enters the context, the model answers, and you want it to mark which source each sentence rests on. The method is plain: number each candidate chunk, write them into the context as [1] text..., have the model annotate with [1], and list sources at the end.

The fixed shape gives each chunk three things: chunkId (the chunk's unique identifier, in the form s07#0), sourceId (which document it belongs to), and sourceTitle (the document's title, for rendering the source list).

What this section is really about is hallucinated citations: the model will cite a number that does not exist. Today's lab holds a real output where, given 8 candidates, it wrote [2][9] in the answer — 9 never existed. That is not rare; it appears fairly consistently in long answers and with many candidates.

Post-processing does two things, neither optional:

1. Strip nonexistent numbers from the answer rather than shipping them. A user opening the source list and finding only 8 concludes the whole system is making things up.
2. Count "this answer contained a hallucinated citation" as a metric (echoing D21's observability dashboard). Stripping is only stanching; a metric is what tells you whether this is getting more or less frequent — its curve turning up usually means retrieval quality or a prompt was recently broken.

Without an evaluation set you cannot even tell that you improved it

On what basis is each of the preceding changes an improvement? Trying a few queries by feel is this field's most common self-deception.

An evaluation set's shape is simple: 20 queries, each labeled by hand with 1 to 3 chunkIds that must be recalled. Note that the labels are on chunks rather than documents — retrieval's granularity is the chunk, and labeling at document level inflates the metric.

Three metrics each answer a different question:

• recall@5: how much of what should be recalled is covered by the 5 that enter the context. This is the number you actually care about, because the model sees only those 5.
• recall@20: how much is covered by the 20 fished up in the first round. It is the ceiling — if recall@20 will not rise, the problem is on the recall side and no amount of reranking helps.
• MRR (mean reciprocal rank): on average, where the first correct result ranks. It is sensitive to ordering quality and breaks ties when recall is equal.

The three tiers' real figures from today's lab are most persuasive read together:

vector only                         recall@5 57%  recall@20 83%  MRR 0.677
vector + keyword + RRF              recall@5 80%  recall@20 95%  MRR 0.732
vector + keyword + RRF + rerank     recall@5 91%  recall@20 98%  MRR 0.908

Read those three rows across and also down. Across: adding the keyword road took recall@5 from 57% to 80% — those 23 points are the vector road's blind spot. Down the second column: recall@20 rose from 83% to 95%, so the ceiling was raised; and the rerank row moved recall@20 only from 95% to 98% (reranking recalls nothing new, it only reorders the existing 20), while what it raised is recall@5 (80% to 91%) and MRR (0.732 to 0.908).

That is the point of the three metrics' division of labor: hybrid search raises the ceiling and reranking moves what is under the ceiling into the top five. Neither substitutes for the other, and neither can be summarized by one number.

Three traps not yet mentioned: chunking, count, and concatenation order

Chunking. A fixed 512 characters with 64 of overlap is a sufficient starting point (today's lab cuts 54 documents into 64 chunks, the longest 509 characters). The overlap prevents a complete idea being cut in the middle — a refund rule's first half in chunk 1 and its second in chunk 2 leaves both incomplete and neither recallable. Better is splitting semantically (by heading, by list item), at the price of writing parsing logic and different sources having different structures. Get fixed-length working first, measure it with an evaluation set, and then decide whether semantic splitting is worth it.

Count. More entering the context is not better. Past a point three things worsen together: token cost rises linearly, irrelevant content dilutes attention, and more numbers means more hallucinated citations. Five is an empirical starting point, and the basis for changing it should be the difference between recall@5 and recall@10 — a small difference means adding entries achieves nothing.

Concatenation order. Put the most relevant one last, right next to the question. Because a model's attention over long context is measurably weaker in the middle than at either end (the industry calls it lost-in-the-middle). Since you must pick an end, pick the one nearest the question. That change is one line, costs nothing, and is free gain.

Source Reading

Hands-On Lab

starter/ has 4 exercise points and runs fully offline under MOCK=1 with no Postgres, Docker, or API key — the infrastructure is in-memory and embeddings are generated deterministically from the text, so recall is genuinely computed and changing one query changes the number. To connect real pgvector, docker compose up -d and set DATABASE_URL (host port 5524), and the same retrieval code runs.

1. Run MOCK=1 SELFTEST=1 pnpm start as-is and note the baseline: only check 1 passes of five, and the three recall@5 tiers are 57%, 69%, 69% (the last two cannot rise because neither the keyword road nor reranking is wired up).
2. Implement bigrams (exercise 1) so a CJK query genuinely recalls on the keyword road, turning check 2 green.
3. Change the fusion to RRF (exercise 2) — discard both roads' raw scores and use ranks only — turning check 3 green while recall@5's second tier jumps to 80%.
4. Wire up rerankTopK (exercise 3), completing check 4's progression to 57%, 80%, 91%.
5. Implement stripHallucinatedCitations (exercise 4), stripping nonexistent numbers and counting the metric, turning check 5 green; then look at what that stripped [9] was.

Interview Questions

Today's four questions are in the bank below, weighted toward why hybrid search is necessary, what reranking solves, and how recall is measured. Expand a question and read the analysis before the key points — "the two roads' blind spots do not overlap" in question 1 is the crux, and question 4's evaluation-set design is a very frequent follow-up, so do not skip it.

Checklist and Tomorrow

• Implement hybrid search combining keyword search and vector search
• Add a reranking step to the retrieval result to improve ranking quality
• Design a recall evaluation method that measures whether retrieval recalled what it should have
• Say why the two roads' scores cannot be summed with weights and must be fused by rank with RRF
• Say what each of recall@5, recall@20, and MRR answers
• All 5 acceptance criteria of the lab pass
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D25) puts this backend in front of human eyes for the first time. As of today it triages, splits tasks, reviews itself, reaches out proactively, finds things accurately, and holds off attacks — and you have not once looked at it through a browser. Tomorrow writes a React chat frontend: streaming rendering, visualizing the tool-calling process, and interrupt and retry, two interactions that look simple and actually require the frontend and backend to cooperate to get right. The order is deliberate: get the backend right before presenting it — done the other way, you paper over a pile of backend defects with frontend loading animations.