Dayward AI
中文
Week 2 · D13About 5 hours

Going to Production: Incremental Sync and Deduplication, Permission-Based Filtering, Cache Layering, Tracing, and the Cost-Latency Ledger

One stretch remains before handing the system to real users: documents change, people have different permissions, repeated questions shouldn't be recomputed every time, and when something breaks you need to find which step was slow and expensive. Today finish all four in one pass.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Design a content-fingerprint-based incremental sync scheme that only recomputes the chunks affected by a document change
  2. Build access control into retrieval itself, and explain why filtering only at generation time is wrong
  3. Add tracing and cost metering to the whole pipeline, and locate where latency and spend each concentrate

Twelve days made this system steadily more accurate while it lived inside three assumptions: the corpus is static, everyone sees the same copy, and every question is computed from scratch. Today all three come down. Once you have read this and finished the lab, scroll back to the top and tick off the three goals.

Plain-Language Walkthrough

The collection changes, and recataloguing is too expensive

A library's card catalog is not finished when it is finished. Every week brings new books, lost books removed, and revised editions replacing old ones. If every change meant recataloguing the whole building, the librarian would do nothing else for a career. What actually happens is a stocktake: take a current list of holdings, reconcile it card by card against the catalog, and touch only the cards that disagree.

An index is the same. Your corpus comes from a drive, a ticketing system, an internal wiki, and they change daily. Rebuilding everything nightly is perfectly fine at thirty documents and becomes a large daily payment at thirty thousand — vectorization is billed by token, and a full rebuild buys the entire knowledge base again every day. Worse, during the rebuild the index is half new and half old, and a question asked then gets an unreproducible result.

Reconciliation handles three kinds of change, none omissible: present in the source and absent from the index is an addition; present in both with differing content is a modification; present in the index and gone from the source is a deletion. Written out it is a dozen lines, and the key is that all three branches must be there:

sync.js
// Three-way reconciliation: compare the source's full set against the index's, touching only disagreements
export function planSync(indexed, incoming) {
  const plan = { added: [], modified: [], unchanged: [], deleted: [] }
  for (const doc of incoming) {
    const known = indexed.get(doc.id)
    if (!known) plan.added.push(doc.id)
    else if (known.contentHash !== contentHash(doc.raw)) plan.modified.push(doc.id)
    else plan.unchanged.push(doc.id)
  }
  // Deletion must iterate the index instead: a source never produces a "this one is gone" record
  const incomingIds = new Set(incoming.map((d) => d.id))
  for (const docId of indexed.keys()) {
    if (!incomingIds.has(docId)) plan.deleted.push(docId)
  }
  return plan
}

In today's lab that reconciliation recognizes one addition, one modification, and one deletion, and vectorizes only 6 chunks; a full rebuild vectorizes 151. Same corpus, a bill twenty-odd times apart.

Content fingerprints: one number answering whether a document needs recomputing

Every row of the reconciliation answers whether the content changed. Comparing full text is slow and bulky, and the general practice is a content hash per document: feed the whole text to sha256 and take the first 16 characters. Change one character and the fingerprint differs entirely; change nothing and it matches exactly.

There is a trap here, already walked into on day three: newlines must be normalized before hashing. The same file uploaded once from Windows and once from a Mac has different byte streams (one with \r\n and one with \n) and identical content. Without normalization every cross-platform re-upload is judged changed, triggering a pointless recomputation. What makes that bug painful is that it raises no error and merely makes your bill look like a full rebuild:

chunk.js
import { createHash } from 'node:crypto'
 
// Normalize before hashing, not as a repair after comparing
export function contentHash(raw) {
  const normalized = raw.replace(/\r\n/g, '\n').trim()
  return createHash('sha256').update(normalized, 'utf8').digest('hex').slice(0, 16)
}

Fingerprints also solve deduplication in passing. The same release notes posted on the wiki by a product manager, saved into a ticket attachment by support, and converted to PDF onto the drive by somebody else give your knowledge base three identical documents. They crowd into results together, taking two of three precious slots. The practice: compute the change-detection fingerprint over the whole file (a changed title or department label must also recompute), and the deduplication fingerprint over the body (stripping filenames and paths and other source-bound metadata, so another copy of the same content is recognizable). One function, two input scopes, and the distinction is deliberate — deduplicating on the whole file never recognizes a duplicate, and detecting change on the body alone misses a changed department label.

Permissions push down into retrieval rather than filtering at generation

Picture looking something up: you walk into the archive, the archivist spreads every file on the table for you to pick from, you pick nine, and they pull three back saying you may not see those. The problem is that you already saw the titles.

Filtering permissions at generation time is that gesture. Even with none of those three in the final answer, the unauthorized documents already entered retrieval, participated in ranking, were read into memory by the program, and most likely were written to a log. The more practical consequence is diluted results: you take the top 8, three of which this user may not see, so after filtering they hold 5, when the legitimate results at ranks 9 and 10 should have moved up — the user feels the system cannot find anything, and your log shows a normal retrieval.

The right approach is one sentence: put the permission condition, the ordering, and the LIMIT in one query. The database prunes rows by predicate first and then orders and takes the top k, with unauthorized rows never once compared:

SQLSQL
SELECT c.chunk_id, c.text, d.department,
       e.embedding <=> $1::vector AS distance
  FROM chunk_embeddings e
  JOIN chunks c    ON c.chunk_id = e.chunk_id
  JOIN documents d ON d.doc_id   = c.doc_id
 WHERE d.department = ANY($2)          -- the permission predicate and the ordering in one statement
 ORDER BY e.embedding <=> $1::vector, c.chunk_id
 LIMIT 8

That is row-level filtering: one index plus a predicate. Its cost is that the whole table passes the predicate to produce those 26 legitimate rows. The alternative is index isolation: build an index per department (or per tenant) and query the matching one directly. Today's lab runs both side by side: the top 8 are identical, and row-level filtering "touched 149 rows" while index isolation touched only 26.

How to choose? By the number and stability of the isolation boundaries. Four departments with near-fixed boundaries make isolation worthwhile; tens of thousands of users each with private documents would crush operations under tens of thousands of indexes, leaving row-level filtering. And one easily overlooked reason: sharing one approximate nearest neighbor index means a tenant with a lot of data genuinely degrades others' retrieval quality — covered on day five.

Caching comes in three layers with entirely different invalidation conditions

The same question should not be recomputed every time. But caching in RAG is not one thing but three, with lifetimes orders of magnitude apart:

LayerWhat is cachedWhat the key must containWhen it invalidates
L1 answerQuestion to final answerQuestion, permission scope, index version, model and prompt versionAny of the above changed
L2 retrievalQuery to hit chunk listQuestion, permission scope, topK, index version, vector backendThe index or a parameter changed
L3 vectorText to vectorText, vector backendOnly a model change

See the pattern? When must this invalidate is equivalent to whether that thing is in the key. Leave one out of the key and its change does not invalidate the cache.

Today's lab's most instructive item is that error case. The first version's answer cache key held only the question — almost everybody writes it that way first, because the same question obviously has the same answer sounds self-evident. Then incremental sync changed doc-006's single-file limit from 200 MB to 500 MB, the index updated, and asking the same question again has the system cheerfully producing 200 MB out of the cache. No error, and a handsome cache hit in the log. Worse is a second phenomenon: an hr-department user asking the same thing hits the same key and gets the product user's answer — an unauthorized disclosure with no trace whatsoever.

The fix is adding the index version and the permission scope to the key:

cache.js
// A sync that genuinely changed the index bumps the version; old keys can never be computed again
export function answerCacheKey({ question, departments, indexVersion, model, promptVersion }) {
  return digest([
    'answer',
    question,
    departments ? [...departments].sort().join(',') : '*', // sorted, so the same scope gives the same key
    indexVersion,
    model,
    promptVersion,
  ])
}

Invalidating by version is far more reliable than precisely deleting the affected cache entries — the latter requires listing which questions this change affected, and that list cannot be produced.

Tracing: take one question apart and see how long each stretch took

When a system is slow, the word slow is worthless. What you need is which stretch is slow. So open a span for each step of a question: record the name, the elapsed time, and the tokens in and out, and print a table when it finishes.

There is no hurry to adopt a platform like Langfuse or OpenTelemetry. They solve how thousands of traces are aggregated and searched, and adopting one before you know which fields a trace should record yields a pile of semantically empty timestamps. In today's lab that recorder is 80 lines and its output looks like this (run offline, with the generation stretch making no network call):

TextText
   —— trace for one question ——
   embed        0.09 ms (  0.9%)      13 in /    0 out  $0.00000026 (  0.0%)
   retrieve     1.20 ms ( 12.5%)       0 in /    0 out  $0.00000000 (  0.0%)
   rerank       4.59 ms ( 47.8%)     690 in /    0 out  $0.00006900 (  3.1%)
   generate     3.73 ms ( 38.8%)     532 in /   38 out  $0.00216600 ( 96.9%)
   total        9.61 ms                                 $0.00223526

That table's point is that time and money get their own columns. In this run the slowest is reranking and the most expensive is generation, and they are not the same stretch — watch only latency and you dive into reranking while leaving the most expensive item untouched in production. Look at them separately before deciding what to optimize.

One caution: the latency distribution offline does not represent a real system. Generation takes milliseconds here because it sends no request; with a real model attached it usually takes over ninety percent of the chain. What is genuinely trustworthy in that record is each stretch's token count — those were really counted.

The cost ledger: what share each of four items takes, and which to optimize first

With token counts, converting to money is one step away. The four are billed in entirely different ways, which is why the ledger is computed separately:

trace.js
// Prices are in dollars per million tokens, so divide by 1e6, not 1e3
export function estimateCost(stage, tokensIn, tokensOut) {
  const price = PRICES[stage]
  return (tokensIn * price.in + tokensOut * price.out) / 1_000_000
}

Vectorization costs money only at ingestion, with a query side of a dozen tokens that can be ignored; but it is a large one-time expense, and a full rebuild rebuys the entire knowledge base — precisely the money incremental sync saves. Retrieval is not billed by token and spends machine time, charged to the server bill. Reranking is billed on the candidate text sent in, which is why it applies only to the top 20 after fusion; reranking the whole store is catastrophic. Generation is billed on both input and output, and the input holds every piece of context you stuffed in — it is almost inevitably the most expensive of the four.

So the optimization order is clear: cut the generation side's input tokens first (fewer chunks, smaller chunks, reranking accurate enough to lower topK), then consider a cheaper model, and only then reranking and retrieval.

Changing the embedding model: dual write, canary, and rollback at any time

One last thing to settle before production: one day you will want to change the embedding model. A new one is more accurate or cheaper, or the old one is being retired.

The trouble is that vectors from different models are not comparable. Dimensions may differ, and even at the same dimensionality the coordinate systems are entirely unrelated — encode a question with the new model, compare distances against documents encoded with the old, and the similarity is pure noise. So changing model is essentially revectorizing the whole knowledge base, that is, a full rebuild. Model selection itself was covered on day two; today covers only doing that rebuild without downtime.

The standard practice is dual write and switch, in four steps.

First add a column to the vector table (embedding_v2, say), nullable. Then run a background job slowly backfilling new vectors, writing only the new column and not touching one byte of the old — production still reads the old column and users notice nothing. With the backfill complete comes the canary: switch a small share of traffic's queries to the new column while running day eight's golden set against both columns and comparing recall and faithfulness. With the numbers holding up, switch all traffic, and delete the old column only after a week or two of observation.

That procedure's whole value is in rollback cost: switching is one configuration entry (which column to read), so reverting on a problem takes a second rather than rerunning an eight-hour rebuild. The lab's tenth item is that chain's minimal version: backfilling 149 v2 vectors while v1 stays queryable throughout, and ranking still working after switching to v2.

Incidentally, this dual-write skeleton is not only for model changes. Changing the chunking strategy, changing how contextual headers are written, adding metadata to chunks — anything that changes the index wholesale is the same procedure. Making it reusable saves far more effort than improvising each time.

Source Reading

Hands-On Lab

🧪 D13 lab: an index service with content-fingerprint incremental sync, access control filtering, multi-level caching, and tracing

Code location: labs/rag-14days/day-13-incremental-and-acl

Acceptance criteria:

  1. Running offline once, the solution passes all ten items with exit code 0; the starter as-is fails five, matching the five exercise points exactly.
  2. The reconciliation item recognizes the added doc-031, the modified doc-006, the deleted doc-030, and the duplicate doc-032, while doc-005 with only its newlines changed to \r\n is still judged unchanged.
  3. This sync vectorizes 6 chunks against a full rebuild's 151, a 96.0% saving; the deleted doc-030 vanishes from the index entirely, with chunk count equal to vector count.
  4. Restricting to support takes the compared rows from 149 down to 26; row-level filtering and index isolation give an identical top 8, with the former touching 149 rows and the latter 26.
  5. The caching item first reproduces both the stale answer (the index says 500 MB and the cache answers 200 MB) and the cross-permission answer leak, and both vanish once the key is completed.

Today's code volume is modest and every exercise point maps to a class of production incident, so copy the self-test output into your notes afterwards — asked in an interview how you guarantee a deleted document never appears, an answer with concrete numbers is in a different league from reciting concepts. The whole path runs without Docker and without a key: with no connection string the index falls back to the in-memory implementation with semantics matching the SQL version, and both sides give item-for-item identical results across the ten checks.

  1. Complete the content fingerprint's normalization and watch the reconciliation item take doc-005 from misjudged as modified to still judged unchanged.
  2. Complete deletion detection, watch doc-030 vanish from the index, and confirm chunk count equals vector count again.
  3. Move permission filtering from after ordering to before it, and watch the compared rows fall from 149 to 26.
  4. Complete the answer cache key so the stale answer and the cross-permission leak both vanish.
  5. Complete the cost conversion for a trace record with all four stretches non-zero, find which is slowest and which is most expensive, then bring up the containerized database and confirm all ten results match the in-memory version.

Interview Questions

Today's 4 questions are in the question bank below, weighted toward incremental update correctness, where permission filtering belongs, and cache invalidation and cost attribution. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

  • Design a content-fingerprint-based incremental sync scheme that only recomputes the chunks affected by a document change
  • Build access control into retrieval itself, and explain why filtering only at generation time is wrong
  • Add tracing and cost metering to the whole pipeline, and locate where latency and spend each concentrate
  • Say what each of the three cache layers' keys must contain, and why the vector cache must not bind the index version
  • Describe the four steps of a dual-write embedding model switch, and why that procedure's value lies in rollback cost
  • All 5 acceptance criteria of the lab pass
  • Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D14) closes out: assemble fourteen days into a multi-tenant, citing enterprise knowledge base question answering with an evaluation panel, and compress the whole course into one decision map. Today has to come before tomorrow — multi-tenancy's foundation is today's permission push-down, and judging whether a scheme is worth adopting rests on today's cost and latency ledger. Without them, that capstone is only a larger toy.

Interview questions

  • After a document changes, how do you recompute only the affected chunks? And how do you guarantee a deleted document really disappears from the index?文档更新之后，你怎么做到只重算受影响的块？被删掉的文档又怎么保证一定从索引里消失？
    Common in ChinaCommon overseasIntermediate#incremental-sync#content-hash#index-maintenance

    How to reason about it · think before answering

    1. There are two halves here and the second one separates candidates. Almost everyone can say 'hash it and compare'; the score comes from bringing up deletion yourself, because it is the one asymmetric case in the whole mechanism.
    2. Give the skeleton first: a three-way reconciliation between the full set from the source and the full set in the index. In source but not indexed is an add; in both but with different content hashes is a modify; indexed but absent from the source is a delete. A modify must replace the document wholesale, deleting old chunks before writing new ones, otherwise a shortened document leaves a tail behind in the index.
    3. Then the fingerprint itself, which is where points are won: sha256 truncated, but normalize line endings and trim before hashing. The same file uploaded from Windows and from macOS differs byte-wise but not in content; skip normalization and every re-upload counts as a change, which is a full rebuild in disguise. It never raises an error, it only shows up on the bill.
    4. The key insight in the second half: a deletion is not an event, it is an absence. Change feeds tell you what changed; nobody ever sends 'I no longer exist'. So deletion detection has to run in the opposite direction — walk the index and find ids the source no longer has. A synchronizer that only listens to change events will wait forever.
    5. At the storage layer, cascade the foreign keys across documents, chunks and embeddings so deleting a document is a single statement and the database does the rest. Hand-written three-step deletes eventually miss one, and the one they miss is a ghost in the index. Close with a verifiable invariant: chunk count must equal embedding count, and a mismatch means orphans.
    6. Expected follow-up: what if the source system itself is unreliable and a pull comes back incomplete? Make pull completeness a precondition for deletion: on a partial pull, apply adds and modifies only, or one failed fetch wipes half your index. Also soft-delete with a retention window so a mistake is recoverable.

    分析过程 · 先想清楚再作答

    1. 这题有两半，区分度全在后半。前半几乎人人答得出「算个哈希比一比」，能不能拿到分取决于你有没有主动讲删除——那是同一套机制里唯一不对称的一种变更。
    2. 先给增量的骨架：拿来源的全集和索引的全集做三向对账。来源有、索引没有是新增；两边都有但内容指纹不同是修改；索引有、来源没有是删除。修改的处理是整篇替换，先删旧块再写新块，不能只追加——不然改短了的文档会在索引里留下一截尾巴。
    3. 接着讲指纹本身，这是给分点：sha256 取前若干位，但**算之前必须先做换行归一化再去首尾空白**。同一份文件从 Windows 传一次、从 Mac 传一次，字节不同内容相同，不归一化就每次都判成变了，等于天天在做全量重建。这个 bug 不报错，只体现在账单上。
    4. 然后是删除这一半的关键判断：**删除不是一个事件，是一个缺席**。文件变动类的通知只告诉你哪些东西变了，永远不会有人发一条「我不存在了」。所以删除检测必须反着来——遍历索引，找出来源里已经没有的 id。只监听变更事件的同步器永远等不到这条消息。
    5. 落到存储上：文档、块、向量三张表用外键级联删除，删文档只写一条语句，剩下的交给数据库。手写三条删除的版本迟早会漏掉一条，而漏掉的那条就是索引里的幽灵。收尾时报一个可验证的指标：块数与向量数必须相等，不等就说明有孤儿。
    6. 可预期的追问：来源系统本身就不可靠、拉不全怎么办？那就把「本次拉取是否完整」当成删除检测的前置条件——拉取不完整时只做新增和修改，不做删除，否则一次拉取失败会把半个索引清空。另外给删除加软删标记和保留期，误删还能回滚。

    Key points

    • Three-way reconciliation covering adds, modifies and deletes; a modify replaces the whole document, old chunks first.
    • Normalize line endings and trim before hashing, or cross-platform re-uploads look like edits and you are doing a full rebuild every night.
    • Deletion is an absence, not an event: walk the index for ids the source no longer has instead of waiting on a change feed.
    • Cascade deletes from documents to chunks to embeddings so one statement suffices; assert chunk count equals embedding count to catch orphans.
    • On an incomplete pull, apply adds and modifies only, and soft-delete with a retention window so mistakes are reversible.

    答题要点

    • 三向对账：新增、修改、删除，缺一不可；修改是整篇替换，先删旧块再写新块。
    • 内容指纹算之前必须先做换行归一化再 trim，否则跨系统重传会被误判为修改，等于天天全量重建。
    • 删除是缺席不是事件，必须反过来遍历索引找出来源里已消失的 id，不能只监听变更通知。
    • 文档、块、向量用外键级联删除，删文档只写一条语句；用「块数等于向量数」当可验证的收尾指标。
    • 来源拉取不完整时只做新增与修改、跳过删除，并给删除加软删与保留期以便回滚。
  • Why can't access control be applied at the generation step? What exactly leaks if you put it there?为什么权限过滤不能放在生成阶段做？放在那里会泄露什么？
    Common in ChinaCommon overseasDeep dive#access-control#filter-pushdown#multi-tenancy

    How to reason about it · think before answering

    1. This checks whether you think about RAG as a system. 'Because it's insecure' scores nothing; the interviewer wants what specifically leaks, and what else goes wrong besides the leak.
    2. Anchor the position with an image: the archivist spreads every file on the table, you pick nine, and only then does he pull three back saying you may not read those. You have already seen the titles. Filtering at generation time is that gesture.
    3. Then split the consequences, and note the second one is what shows engineering experience. First, exposure: the unauthorized documents were retrieved, ranked, read into process memory, and almost certainly written to retrieval logs and traces, even if none of their text reaches the answer. Second, dilution: you take the top 8, three are off-limits, the user gets five, and the legitimate results ranked ninth and tenth never get promoted. The user experiences 'it can't find anything' while your logs show a perfectly normal retrieval.
    4. State the fix: put the permission predicate in the same query as the ordering and the LIMIT, so the database prunes rows before ranking and unauthorized vectors are never compared. Cover both shapes: row-level filtering is one index plus a predicate; index isolation is a separate index per boundary.
    5. Give the selection criterion: the number and stability of the isolation boundaries. A handful of departments that rarely change makes isolation worthwhile; tens of thousands of per-user private document sets leave you with row-level filtering, because that many indexes is unmanageable. Add the shared-index side effect: a large tenant degrades everyone else's retrieval quality because candidate slots are shared.
    6. Expected follow-up: what about caching? It is the same bug's second crime scene. The answer cache key must include the permission scope, or one user's answer will be served to another, and that leak leaves no trace in the retrieval log at all.

    分析过程 · 先想清楚再作答

    1. 这题在考你有没有真的把 RAG 当系统看。答成「因为不安全」拿不到分，面试官要的是「具体泄露了什么」和「除了泄露还有什么后果」两件事。
    2. 先用一个画面把位置说清楚：档案管理员先把全部档案摊在桌上让你挑，你挑完他再抽走三份说这些不能看——你已经看见标题了。在生成阶段过滤就是这个动作。
    3. 然后拆后果，两条，第二条更能显出做过工程：一是**泄露面**，越权文档已经进过检索、参与过排序、被进程读进过内存、大概率写进了检索日志和链路追踪，哪怕最终答案里没有它的内容；二是**结果被稀释**，取前 8 条里有 3 条不该看，筛掉只剩 5 条，而本该补位的第 9、10 名合法结果永远没机会上来——用户体感是「查不到」，你的日志里却是一次正常检索。
    4. 给正确做法：把权限谓词和排序、LIMIT 写进同一条查询，数据库先裁行再排序取前 k，越权的行一次都没被比较过。两种落法要都讲：行级过滤是一份索引加一个谓词，索引隔离是按边界各建各的索引。
    5. 选型判据要给出来：看隔离边界的数量和稳定性。部门这种个位数且几乎不变的边界，隔离划算；几万个用户各自的私有文档就只能行级过滤，否则运维扛不住。补一句共用索引的副作用——数据量大的租户会拖慢别人的检索质量，因为候选名额是共享的。
    6. 可预期的追问：缓存怎么办？这是同一个问题的第二现场——答案缓存的 key 里必须带上权限范围，否则一个用户的答案会被另一个用户命中，而且这条泄露路径连检索日志都不会留下痕迹。

    Key points

    • Filtering at generation time means unauthorized documents were already retrieved, ranked, held in memory and written to logs and traces; the exposure is far wider than 'did the text reach the answer'.
    • The second consequence is dilution: filtered-out slots are not backfilled, so users see 'nothing found' while the log shows a normal retrieval.
    • The fix is to put the permission predicate in the same statement as ordering and LIMIT so the database prunes before ranking.
    • Choose between row-level filtering and index isolation by the count and stability of the boundaries; a shared index lets a large tenant crowd out a small one's candidate slots.
    • Caching is the same bug's second crime scene: the answer cache key must carry the permission scope or answers leak across users without a trace.

    答题要点

    • 在生成阶段过滤时，越权文档已经被检索、排序、读进内存并写进日志与追踪，泄露面比「答案里有没有」大得多。
    • 第二个后果是结果被稀释：筛掉之后名额空着不补，用户体感是查不到，日志里却是一次正常检索。
    • 正确做法是把权限谓词和排序、LIMIT 写进同一条查询，让数据库先裁行再排序取前 k。
    • 行级过滤与索引隔离的选型判据是隔离边界的数量与稳定性；共用索引时大租户会挤占小租户的候选名额。
    • 缓存是同一个漏洞的第二现场：答案缓存的 key 必须包含权限范围，否则会跨用户串答案且不留痕迹。
  • What can be cached in a RAG system, and what are the invalidation conditions for each?RAG 系统里有哪些东西可以缓存？各自的失效条件是什么？
    Common in ChinaCommon overseasIntermediate#caching#invalidation#cost-optimization

    How to reason about it · think before answering

    1. This looks like a giveaway and is actually a filter. 'Cache the question and answer' earns a third of the credit; the interviewer is waiting for the layering and the per-layer invalidation rules.
    2. Lead with a transferable rule: 'when must this be invalidated' is the same question as 'is that thing part of the key'. Leave something out of the key and changes to it will never invalidate the entry. With that rule the three layers derive themselves.
    3. Then go layer by layer. The answer layer maps a question to a final answer; its key needs the question, the permission scope, the index version, and the model plus prompt version. The retrieval layer maps a query to a hit list; its key needs the question, scope, topK, index version and embedding backend, but not the generation model. The embedding layer maps text to a vector; its key is just the text and the backend.
    4. Emphasize the counterintuitive part of the embedding layer: it is content-addressed, so the index version must not be in its key. Put it there and a single sync invalidates tens of thousands of vectors, which is exactly the full rebuild you added caching to avoid. This is the one layer that can live a long time, even on disk.
    5. Offer a concrete invalidation mechanism: version numbers rather than targeted deletion. Bump an index version whenever a sync actually changes something and old keys simply stop being computed. Targeted deletion would require enumerating which questions a change affected, and that list cannot be produced.
    6. Expected follow-up: can you give a real 'should have expired but didn't' case? Yes: an answer cache keyed only on the question. A document's limit changes from 200 MB to 500 MB, the index is updated, and the same question still returns 200 MB. Nothing errors; the log shows a clean cache hit. The same key also serves one department's answer to a user from another.

    分析过程 · 先想清楚再作答

    1. 这题看起来是送分题，实际是筛人题。答成「把问答结果缓存起来」只拿到三分之一，面试官等着听的是「分几层」和「各自什么时候失效」。
    2. 先给一条能迁移到别的题上的判断依据：**「什么时候必须失效」这个问题，等价于「key 里有没有把那样东西算进去」。** key 少放一样，那样东西变了缓存就不会失效。有了这条，三层的答案自己就长出来了。
    3. 然后逐层给：答案层缓存问题到最终答案，key 要有问题、权限范围、索引版本、模型与提示词版本；检索层缓存检索式到命中块列表，key 要有问题、权限范围、topK、索引版本、向量后端，但不需要模型；向量层缓存文本到向量，key 只有文本和向量后端。
    4. 重点讲向量层的反直觉之处：它是**内容寻址**的，文本没变、模型没变，向量就不会变，所以**不能把索引版本放进它的 key**。放进去的话一次同步就作废几万条向量，正好绕回全量重建——你加缓存想省的那笔钱又花回去了。这一层可以放很久甚至持久化。
    5. 给一个具体的失效手法：用**索引版本号**而不是精确删除。同步只要真的改动了索引就把版本号加一，旧 key 再也算不出来，自然没人读得到。精确删除要求你能列出「这次改动影响了哪些问题」，而那是列不出来的。
    6. 可预期的追问：能举一个「该失效却没失效」的真实例子吗？答：答案缓存的 key 只放了问题本身，文档里的上限从 200 MB 改成 500 MB、索引已经更新，再问同一个问题仍然返回 200 MB。它不报错，日志上是一次漂亮的缓存命中；同一个 key 还会让另一个部门的用户直接命中别人的答案。

    Key points

    • Three layers — answer, retrieval, embedding — with lifetimes orders of magnitude apart; treating them as one thing is the mistake.
    • The rule is that 'when must it expire' equals 'is it in the key'; anything left out of the key can never invalidate the entry.
    • The answer key carries question, permission scope, index version, model and prompt version; the retrieval key drops the model and adds topK and the embedding backend.
    • The embedding layer is content-addressed and keyed only on text plus backend; adding an index version turns every sync back into a full rebuild.
    • Version-based invalidation beats targeted deletion because you cannot enumerate which questions a given change affected.

    答题要点

    • 分三层：答案、检索、向量，三者的寿命差着数量级，不能当成一件事。
    • 判断依据是「什么时候必须失效」等价于「key 里有没有算进那样东西」，key 少一样就永远失效不了。
    • 答案层 key 要有问题、权限范围、索引版本、模型与提示词版本；检索层去掉模型、加上 topK 与向量后端。
    • 向量层是内容寻址的，key 只有文本与后端；把索引版本放进去会让每次同步都退化成全量重建。
    • 用索引版本号做失效比精确删除可靠，因为「这次改动影响了哪些问题」根本列不出来。
  • You need to switch embedding models. How do you migrate a live system without downtime and without losing recall?要换一个 embedding 模型，线上系统怎么迁移才能不停机也不掉召回？
    Common in ChinaCommon overseasDeep dive#embedding-migration#zero-downtime#rollout

    How to reason about it · think before answering

    1. The crux is why you cannot swap in place. Jumping straight to the steps without establishing that reads like reciting a runbook.
    2. Set up the premise: vectors from different models are not comparable. Dimensions may differ, and even at equal dimensions the coordinate spaces are unrelated, so encoding the query with the new model and comparing against documents encoded with the old one yields noise. Switching models therefore means re-embedding the entire corpus.
    3. Then the four steps: add a nullable second vector column; backfill it with a background job while the old column is untouched and still serves live traffic; canary a slice of traffic onto the new column while running the golden set against both columns to compare recall and faithfulness; cut over fully once the numbers hold, and drop the old column only after a week or two of observation.
    4. Name the payoff explicitly, because this is where the points are: the value of the whole procedure is the rollback cost. Cutover is a config change naming which column to read, so reverting takes a second rather than re-running an eight-hour rebuild. A migration plan with no rollback path is not a plan.
    5. Add two engineering details: build the approximate-nearest-neighbor index on the new column after the backfill, not during it, since concurrent building is slow and prone to locking; and make the backfill resumable and rate-limited, or it will exhaust the embedding API quota and drag live queries down with it.
    6. Expected follow-up: how do you prove the new model is actually better? Not from an offline metric alone — run an A/B on the same golden set with identical retrieval parameters and report four numbers: recall, faithfulness, latency and cost. A conclusion resting on the first number only does not hold. Note also that switching models is the one moment when the embedding cache genuinely must be invalidated.

    分析过程 · 先想清楚再作答

    1. 这题的题眼是「为什么不能就地换」。没有先说清这一点就直接讲步骤，会显得是在背流程。
    2. 先给前提：不同模型的向量之间**没有可比性**。维度可能不同，即使维度相同坐标系也完全不是一回事，用新模型编码问题去和旧模型编码的文档比距离，算出来的相似度是纯噪声。所以「换模型」实质上等于「把整个知识库重新向量化一遍」。
    3. 然后给四步：加一列新向量、允许为空；后台任务慢慢回填新列，旧列一个字节不动，线上仍走旧列；小流量灰度到新列，同时用标准答案集在两列上各跑一遍比召回率与忠实度；数字站得住再全量切换，旧列观察一两周后才删。
    4. 把这套流程的价值点破，这是给分点：**它的价值全在回滚成本上**。切换只是改一个配置项「走哪一列」，出问题时切回去是一秒钟的事，而不是重跑一遍八小时的重建任务。凡是拿不出回滚路径的迁移方案都不算方案。
    5. 补两个工程细节：新列的近似最近邻索引要在回填完之后再建，边写边建又慢又容易锁表；回填要能断点续传并限速，否则会把 embedding 接口的配额打满，把线上查询一起拖垮。
    6. 可预期的追问：怎么证明新模型确实更好？答：不能只看离线指标涨没涨，要在同一份标准答案集、同一套检索参数下跑 A/B，报召回率、忠实度、延迟、花费四笔账；只报第一笔的结论不成立。另外注意换模型会让缓存里的向量全部作废，那是这次迁移唯一该作废向量缓存的时刻。

    Key points

    • Vectors from different models are not comparable, so a model switch is equivalent to re-embedding the entire corpus.
    • Four steps: add a nullable second vector column, backfill in the background, canary with the golden set scored on both columns, then cut over once the numbers hold.
    • The whole value lies in rollback cost: cutover is a config change, so reverting takes a second instead of another full rebuild.
    • Build the ANN index on the new column after the backfill; make the backfill resumable and rate-limited so it does not exhaust the embedding quota and stall live queries.
    • Validate with an A/B on one golden set reporting recall, faithfulness, latency and cost; a model switch is also the only time the embedding cache truly must be invalidated.

    答题要点

    • 不同模型的向量之间没有可比性，所以换模型等价于把整个知识库重新向量化一遍。
    • 四步：加一列可空的新向量、后台回填、小流量灰度并用标准答案集在两列上对比、数字站得住再全量切换。
    • 这套流程的价值全在回滚成本上：切换是改一个配置项，回滚是一秒钟的事而不是重跑一次重建。
    • 新列的近似最近邻索引在回填完成后再建；回填要可断点续传并限速，别把接口配额打满拖垮线上查询。
    • 验证要在同一份标准答案集上跑 A/B，同时报召回率、忠实度、延迟与花费四笔账；换模型也是唯一该作废向量缓存的时刻。

Comments