Week One Capstone: Assembling Six Days of Parts Into a One-Command Question-Answering Service, and a Retrospective

Two pipelines, one junction

With the cuts drawn, the next question is what the ingestion and query pipelines share.

Ingestion is batch: tens of seconds to parse, chunk, vectorize, and write thirty documents, and a failure means rerunning. A query is an online request: a result within a few hundred milliseconds, and a failure is seen by the user on the spot. Their error handling and timeout strategies are simply not the same thing.

The easiest mistake is seeing that both need embedding, extracting a shared module, and then finding that ingestion wants batched retries while queries want fast failure, with the module filling up with if (isIngest). What is shared should be an interface, not a flow. In this version the two pipelines share exactly one thing: the storage layer's interface. Ingestion writes, queries read, and no business code is shared beyond that.

Ingestion pipeline                Query pipeline
loadDocs   ← parsing (D3)         retrieve   ← two paths (D1 + D2)
  ↓                                 ↑
chunker    ← chunking (D4)        Store  ←──┘
  ↓                                 ↓
embed      ← vectorizing (D2)     answer     ← assembly + cited generation (D6)
  ↓
Store ─────────────────────────────┘

embed appears on both pipelines, and it is two calls to one interface, not a shared flow. There is a hard constraint here: vectors for chunks and vectors for questions must come from the same model. Querying an index built with model A using model B's question vectors makes the distances meaningless, raises no error, and quietly hands you a pile of garbage that looks like results. So every vector records which model computed it (the chunk_embeddings.model column), the only way to discover this class of accident afterwards.

The storage interface is only a few lines and is the thing in the whole service most worth settling first:

export interface Store {
  init(): Promise<void> // a repeatable migration
  // Replace one document and its chunks in one go. It must be a transaction:
  // deleting old chunks without writing new ones erases the document from the store
  replaceDoc(doc, chunks, vectors, model): Promise<void>
  listChunks(): Promise<StoredChunk[]>
  getChunk(chunkId: string): Promise<StoredChunk | null>
  vectorSearch(query: number[], topK: number): Promise<VectorHit[]>
}
 
// The only selection point: a connection string means the real database, otherwise the in-memory one.
// Code outside this interface has no idea who it is talking to
export function createStore(): Store {
  return process.env.DATABASE_URL ? new PgStore(process.env.DATABASE_URL) : new MemoryStore()
}

class Store(Protocol):
    async def init(self) -> None: ...  # a repeatable migration
 
    # Replace one document and its chunks in one go. It must be a transaction:
    # deleting old chunks without writing new ones erases the document from the store
    async def replace_doc(self, doc, chunks, vectors, model: str) -> None: ...
    async def list_chunks(self) -> list[StoredChunk]: ...
    async def get_chunk(self, chunk_id: str) -> StoredChunk | None: ...
    async def vector_search(self, query: list[float], top_k: int) -> list[VectorHit]: ...
 
 
def create_store() -> Store:
    """The only selection point: a connection string means the real database, otherwise in-memory.
    Code outside this protocol has no idea who it is talking to"""
    url = os.environ.get("DATABASE_URL")
    return PgStore(url) if url else MemoryStore()

The in-memory implementation is not a stub pretending writes succeeded; it genuinely reimplements PostgreSQL's semantics — replacing a document's old chunks, ordering by cosine, breaking ties by id — at the cost of a full scan. Its benefit shows in week two: after D8 you run dozens of retrievals a day for evaluation, and if each needs a container up first, you will start skipping them.

Parameters out of the code: how far configuration should go

Week two repeatedly raises the same class of question: 500 characters or 300? Four results or eight? What happens with the vector path off?

Those questions have exactly one correct way of being answered: change one configuration entry, rerun, and read the numbers. If changing the chunk size means editing pipeline.ts and changing the number of paths means editing retrieve.ts, you will try three configurations a day at most and forget what you tried. So this version collects the parameters into one rag.config.json:

{
  "chunking": { "strategy": "heading", "maxChars": 500, "minChars": 60, "overlapChars": 60 },
  "embedding": { "backend": "auto", "model": "text-embedding-3-small", "dimensions": 384 },
  "retrieval": {
    "routes": ["bm25", "vector"],
    "perRouteTopK": 8,
    "topK": 4,
    "weights": { "bm25": 1, "vector": 0.6 },
    "gates": { "bm25MinScore": 6, "vectorMinCosine": 0.2 },
    "bm25": { "k1": 1.2, "b": 0.75 }
  },
  "generation": { "model": "claude-sonnet-5", "maxContextChars": 6000 }
}

Four sections for four swappable parts, showing at a glance which knobs this machine has. Once running, one environment variable overrides temporarily without editing the file: CHUNK_STRATEGY=fixed takes the same corpus from 134 chunks to 52, and ROUTES=bm25 turns the vector path off.

One boundary must be drawn: environment variables override and never define. The configuration file is always the manual, so one file tells you which parameters exist and what their defaults are. Scatter the defaults across a dozen process.env.XXX ?? someNumber expressions instead and nobody can say what configuration the system is running under, and a problem cannot be reproduced.

The part of the configuration most worth stopping at is those two gates: a candidate is reachable if any path's raw score clears its own line, and clearing none means an outright refusal. There is a counter-intuitive pothole here — the gate must sit on the raw score and never on the fused score. The fused score is normalized against this query's top score, so even when the most relevant item in the whole run scores 3, its fused score is still 1.0, and gating on it means the system never refuses again.

As for that 0.2 in vectorMinCosine, honestly: it was measured, not derived. I printed the offline hash vectors' similarity distribution, found genuinely relevant chunks landing just past 0.2 (0.20 to 0.27 measured), and found a question with no answer in the corpus topping out at 0.13, so the line went at 0.2. A real embedding's distribution is entirely different and needs recalibrating; and how to calibrate it correctly has no answer today — which is precisely the first debt.

Three endpoints, none omissible

The service exposes exactly three endpoints, and dropping any one makes it not work:

• POST /ingest: given a directory, parse, chunk, vectorize, and store, returning how many documents were processed and how many chunks were cut.
• POST /ask: given a question, return the cited answer, the chunks used, and the candidates that failed the gates.
• GET /chunks/:chunkId: given a chunk id, return its original text.

The third is the easiest to cut and is precisely the redemption of this whole apparatus. D6 has the model emit only the chunk numbers [1] [2] issued to it this time, code checks that the sentence and that chunk genuinely correspond, and then the number is exchanged for a clickable id such as doc-006#c04 and returned to the user. Without something to click, the whole thing is no different from "as far as I know." A citation's value is not in being labeled but in being checkable.

The response body has another easily cut field: the candidates that failed the gates. It is useless to the user and useful to you — when production produces a "why did it not answer," the first thing to look at is what retrieval actually fetched: nothing fetched at all (an index or tokenization problem), fetched with insufficient score (a gate problem), or a sufficient score the model did not use (a prompt problem). The three failures have entirely different fixes, and without that field you cannot even classify them.

The ingestion endpoint has a detail that must be right. The keyword path's BM25 index lives in memory, so ingestion must rebuild the index afterwards:

app.post('/ingest', async (req) => {
  const report = await ingest(dir, store, cfg, embedder)
  // The keyword index must be refreshed after ingestion, or newly stored chunks are findable by vector
  // and not by keyword. That one-path-works failure is the hardest to diagnose, since it just looks odd
  const indexed = await retriever.refresh()
  return { ...report, indexed } // indexed is returned precisely so this can be verified to have happened
})

@app.post("/ingest")
async def ingest_endpoint(body: IngestBody):
    report = await ingest(body.dir or CORPUS_DIR, store, cfg, embedder)
    # The keyword index must be refreshed after ingestion, or newly stored chunks are findable by
    # vector and not by keyword. That one-path-works failure is the hardest to diagnose
    indexed = await retriever.refresh()
    return {**report, "indexed": indexed}  # returned precisely so this can be verified

One command to start: who runs the migration

One-command startup sounds like an operations concern and in fact decides how many times a day you can experiment. A project needing five minutes and six commands gets run twice a day; one that is just docker compose up -d gets run ten times an hour. Week two rebuilds indexes and reruns evaluations daily, and that gap gets multiplied dozens of times.

The database side needs three things prepared: installing the pgvector extension, creating three tables, and building the vector index. The question is who runs them and when. The least effort is hanging the schema script in the container's initialization directory, executed automatically the first time the database starts:

services:
  postgres:
    image: pgvector/pgvector:pg16
    ports:
      - '5557:5432'
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U rag -d rag']
      interval: 5s
      retries: 20
    volumes:
      - ./migrations:/docker-entrypoint-initdb.d:ro
      - pgdata:/var/lib/postgresql/data

One pothole here is worth remembering separately: scripts in the initialization directory execute once, only when the data volume is empty. Edit the schema statements, restart the container, and not one new statement runs, after which you spend an afternoon on "column does not exist." So the server also runs an equivalent set of statements written entirely with IF NOT EXISTS at startup as a backstop. The two look duplicated and solve two scenarios: a brand-new environment and one with existing data.

healthcheck is not decoration either: several seconds pass between a container having started and being able to accept connections, and without it the service fails to connect and exits during those seconds while you assume a configuration error.

Retrospective: do this week's six decisions still hold?

Now lay this week's decisions out one at a time and ask the same question: on what basis at the time, and does it still hold?

D1 doing pure BM25 with no vectors. The reasons were explainability, zero dependencies, and a baseline. Its value turns out to be larger than expected — today's self-test prints nine questions' results alongside D1's, showing at a glance which hits changed. Optimization without a baseline is all self-congratulation. What needs correcting is the implication that keywords would be superseded by vectors: they were not, and the keyword path still contributes the overwhelming majority of hits today.

D2 using the API embedding on the main line with a local model as fallback. The reason was convenience and still holds, plus one thing not anticipated then: precisely because it might change at any time, it forced that unified interface into existence, and that interface is today the cleanest layer in the service.

D3 unifying parse output into a node sequence with heading paths. This is the highest-return decision and it paid off sooner than expected. Today's heading-based chunking uses headingPath directly, chunk bodies can carry a locating note, and citation display can show which section something came from — all free from that data structure.

D4 choosing a chunking approach by hit rate rather than intuition. This one is awkward today: I chose heading-based cutting and cannot give strict evidence. All that was measured is chunks going from 52 to 134 and five of nine baseline questions changing their hit document sets. Better or worse, I cannot say — that is not evaluation, that is observation. The evidence waits for D8.

D5 staying on PostgreSQL rather than a dedicated vector store. A hundred-odd chunks cannot even use an approximate index, and a sequential scan beats HNSW on both speed and accuracy; the index line still goes into the migration script, because it must already be there the day the volume rises. The preconditions were written out plainly: data volume, write frequency, filter complexity, operational capacity, with any one crossed prompting a reassessment.

D6 citations verified by code rather than trusting the model. Today's self-test deliberately feeds in two bad citations: one with an out-of-range number and one whose number is real and whose content does not match. The former is easy to catch and the latter surfaces only after computing overlap. Its weight comes from being the one rule a prompt cannot cover: a prompt makes the model more inclined not to fabricate, and only code verification guarantees a fabrication does not reach the response.

In one line: five of six decisions hold, and the one that does not was not a wrong choice but a missing scale.

Three debts left for week two

An honest milestone must list what it did not achieve. This version has three debts, each with a definite repayment date:

First: no evaluation, so every configuration was guessed. 500-character chunks, four results, a 0.2 gate, weights of 1 to 0.6 — not one number has evidence. Worse, changing any parameter now leaves me unable to say whether it improved anything, only that the hit documents changed. This is the heaviest debt, because it makes the other two unverifiable. D8 builds a golden set, implements recall and ranking metrics, and measures this configuration's baseline; until then every "I think this is better" is a guess.

Second: the two paths run in parallel with crude fusion. Each is normalized against its own top score and weighted-summed, which is blind to a path whose top score is itself low, and there is no reranking, with the top four decided entirely by the fused score. D9 switches to reciprocal rank fusion (which looks only at rank rather than score, sidestepping the scale problem) followed by a cross-encoder rerank, proving with D8's evaluation how much each step contributed.

Third: no permissions and no incrementality. POST /ingest is a full rerun, fine for thirty documents and hours plus a real vectorization bill for thirty thousand. The content_hash in the documents table has not been read once today; it is groundwork for D13. Permissions are more direct: chunks carry a department and nothing uses it — filtering at generation time means it already leaked, since the material entered the context long before. D13 pushes filtering down into the retrieval query and adds cache layering and tracing.

Two smaller ones recorded here so they are not forgotten: the BM25 index lives in one machine's memory and will be inconsistent across a multi-instance deployment; and ingestion is serial with no concurrency and no retry on failure.

Source Reading

Hands-On Lab

Today's lab is week one's milestone, with two or three times the code of earlier days and very little new logic — most of it is rearranging six days' work behind interfaces. The four exercise points land on chunking, vectorization, retrieval gates, and citation verification, each turning one self-test item from a cross to a tick. The whole path runs without Docker and without an API key: with no connection string, storage falls back to the in-memory implementation, and MOCK=1 swaps vectors for deterministic hashes and generation for template assembly.

1. Arrange the six days' modules along the eight files' boundaries, settling the storage interface first and then filling in the two implementations, and run the self-test to confirm the service starts.
2. Complete the chunking exercise: the ingestion item's chunk count goes from 30 to 134, showing heading-level cutting genuinely took effect.
3. Complete the vectorization and gate exercises: the vector item computes non-zero similarities and the team outing question reaches a refusal.
4. Complete the citation verification exercise: feed in two bad citations and watch both the out-of-range one and the mismatched one appear in the blocked list.
5. Bring up the containerized database, add the connection string, rerun the self-test, confirm the eight results match the in-memory version, and copy the nine lines beside day one's baseline into your own notes.

Interview Questions

Today's 4 questions are in the question bank below, weighted toward module boundaries, configuration and swappability, and a retrospective on all of week one's decisions. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets.

Checklist and Tomorrow

• Split the ingestion and query pipelines into clear module boundaries, and explain why each module can be swapped independently
• Bring the service up along with its database with one command, and run the full path from feeding in documents to getting a cited answer
• List three pieces of technical debt left from this week, and say which day of week two each one gets paid off
• State the criterion for where a cut goes, and use it to explain why the vectorization layer must be swappable
• Explain why a retrieval gate must sit on each path's raw score rather than the fused score
• All 5 acceptance criteria of the lab pass
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D8) we build the scale: produce a twenty-question golden set, implement recall, mean reciprocal rank, and normalized discounted cumulative gain, use a model as judge for faithfulness, and produce a baseline report for today's service. The order is deliberate — an evaluation needs something to evaluate, and today's machine is the object under evaluation. From tomorrow, whether each of this week's debts has been repaid is decided by the numbers.