Vector Indexes and Store Selection: HNSW vs. Inverted File, Quantization to Save Memory, Filtered Queries and Multi-Tenant Isolation
Push vector search from working to holding up under load: understand how the two index structures are built and how their parameters affect recall and latency, compress memory with half-precision and binary quantization, solve the trickiest problem — filtered queries — then give selection criteria between pgvector and a dedicated vector store.
Today's Goals
- Explain how Hierarchical Navigable Small World graphs and inverted files are each built, and what each of their tunable parameters affects
- Explain why filtered vector queries can under-recall, and state when iterative scanning versus pre-filtering each applies
- Give selection criteria for what scale and what constraints should stay on PostgreSQL, and when to move to a dedicated vector store
Yesterday you could cut documents into suitable chunks. Today handles what happens once chunks multiply: at ten thousand or a million chunks, is retrieval still fast, still accurate, and still correct once a condition is added? Once you have read this and finished the lab, scroll back to the top and tick off the three goals.
Plain-Language Walkthrough
Three hundred thousand books cannot be leafed through one by one
The query you wrote on day two essentially computes a distance from the question vector to every row's vector, sorts, and takes the top ten. With a few hundred chunks from thirty documents that is entirely fine and returns in a blink.
But its cost is proportional to the store's total size: ten times the rows, ten times the time. In our lab, twenty thousand rows at 384 dimensions on single-machine Docker gives a median latency of 5.3 milliseconds for one full comparison — tolerable-sounding, yet already six times the 0.8 milliseconds of an index scan, and that line is straight: at two million rows it becomes half a second, which the user experiences as a clear stall.
So from here retrieval changes its objective: no longer certainly finding the nearest ten, but probably finding the nearest ten within an acceptable time. Approaches of that kind are collectively approximate nearest neighbor search. Approximate is today's entire premise — every set of results you get from now on may be missing a few that belonged in it.
The metric for how many are missing is recall: of the true top ten, how many the index's top ten hit. Where does the truth come from? Turn the index off and scan the whole table; that is the ground truth. It is the foundation of all tuning work — without ground truth, every parameter you tune only changes something you cannot see. Today's lab's first exercise point is exactly that: if the query computing the ground truth also uses the index, the recall you measure is permanently 100% and you never notice.
That measuring apparatus is only a dozen lines and produces every table today:
// Ground truth: turn the index off and scan the whole table. Only that yields real recall;
// otherwise you compare the index against itself and the result is permanently near 100%
export async function groundTruth(sql, queryVector, k = 10) {
return sql.begin(async (tx) => {
await tx.unsafe('set local enable_indexscan = off')
await tx.unsafe('set local enable_bitmapscan = off')
const rows = await tx`
select chunk_id from chunk_embeddings
order by embedding <=> ${queryVector}::vector limit ${k}
`
return rows.map((row) => row.chunk_id)
})
}
// Divide by the ground truth's count, never a hard-coded k: with a filter there may be fewer than k rows
export function recallAt(returned, truth) {
if (truth.length === 0) return 1
const truthSet = new Set(truth)
return returned.filter((id) => truthSet.has(id)).length / truth.length
}async def ground_truth(conn, query_vector, k: int = 10) -> list[str]:
"""Ground truth: turn index scans off and compute over the whole table. Using index
results as truth gives permanently near-100% recall"""
async with conn.transaction():
await conn.execute("SET LOCAL enable_indexscan = off")
await conn.execute("SET LOCAL enable_bitmapscan = off")
rows = await conn.fetch(
"SELECT chunk_id FROM chunk_embeddings ORDER BY embedding <=> $1 LIMIT $2",
query_vector,
k,
)
return [row["chunk_id"] for row in rows]
def recall_at(returned: list[str], truth: list[str]) -> float:
"""Divide by the ground truth's count, never a hard-coded k: with a filter there may be fewer rows"""
if not truth:
return 1.0
hit = len(set(returned) & set(truth))
return hit / len(truth)Back to the library. Approximate retrieval admits one thing: you cannot search the whole building to find ten books and must rely on the catalog. And how the catalog is organized decides directly what you will miss.
Interview questions
How do you choose between an HNSW index and an IVFFlat index? Give one scenario that forces each choice, and name the parameter you would tune first in each.分层可导航小世界图和倒排文件索引你会怎么选?各说一个必须选它的场景,以及各自最该调的参数。
Common in ChinaCommon overseasIntermediate#vector-index#hnsw#ivfflat🔒 Unlock this chapter to see the answer keys.
Why does a vector search with a WHERE clause return fewer results than expected, and what are the fixes and their costs?为什么加了 WHERE 条件的向量检索会漏结果?有哪几种修法,代价分别是什么?
Common in ChinaCommon overseasDeep dive#filtering#iterative-scan#recall🔒 Unlock this chapter to see the answer keys.
If you switch your vectors from full precision to half precision or binary quantisation, how do you verify that recall has not dropped materially?把向量从全精度换成半精度或二值量化,你会用什么方法确认召回没有明显下降?
Common in ChinaCommon overseasIntermediate#quantization#evaluation#recall🔒 Unlock this chapter to see the answer keys.
When should you move your vectors out of PostgreSQL into a dedicated vector database? Give measurable triggers, and also make the case for staying.什么时候应该把向量搬出 PostgreSQL?给出可量化的触发条件,也说说不该搬的理由。
Common in ChinaCommon overseasIntermediate#vector-database#architecture#trade-offs🔒 Unlock this chapter to see the answer keys.
