Getting Documents In: Parsing PDF and HTML, Tables and Scans, Cleaning Rules, and Metadata You Must Keep

Today's Goals

1. Name three typical PDF parsing failures — multi-column reordering, tables collapsing into one line, headers and footers leaking into the body — and give a fix for each
2. Design a document metadata structure that gives later citation labeling, permission filtering, and incremental updates something to rely on
3. Write a parsing pipeline that normalizes documents from multiple sources into one intermediate representation, and run an assertable quality check on the parsed result

For two days we experimented on a ready-made, spotless Markdown corpus. Today it goes away, replaced by how documents arrive in the real world: a pile of PDFs, a few help center pages, and one scanned document. Once you have read this and finished the lab, scroll back to the top and tick off the three goals.

Plain-Language Walkthrough

Look at the ingredients before the pan

A cook taking an order does not light the stove first but looks at the ingredients. Are the greens wilted, is the fish fresh, has the salt caked with damp — all of that gets checked before anything hits the pan. Because after it hits the pan, no precision of heat and no refinement of seasoning rescues an unfresh fish. Worse, when the diner tastes the problem, the blame falls on the cook's skill and nobody thinks of the fish.

A retrieval system is that kitchen. D1's index, D2's vectors, and the reranking and query rewriting of the coming eleven days are all heat and seasoning. What sets the ceiling is the ingredients coming into the kitchen — that is, parsing: turning PDFs, web pages, Word files, and scans into a searchable stream of text.

When that step breaks, the symptoms are extremely hard to recognize. It raises no error and does not crash; it quietly puts a row-crossed table into the index, a rule cut off mid-sentence, a line of "internal, do not distribute" that appears on every page. Then one day a user asks what the professional plan's storage quota is and the system confidently answers "unlimited" — because that table lost a column separator during parsing and "unlimited" was the value in the neighboring column.

Return to D1's five-stage diagram: chunking, indexing, retrieval, context assembly, generation. Parsing precedes all five as stage zero. Its errors get amplified by every later stage — chunking cuts along wrong text boundaries, indexing records wrong terms in the inverted list, retrieval confidently ranks it first, and generation cites it earnestly. That is the concrete transmission chain behind "parsing quality sets retrieval quality's ceiling", and reciting that chain is enough when an interview asks.

The three mountains of PDF

PDF is the most common and hardest format in an enterprise knowledge base. Hard because of one counter-intuitive fact:

A PDF file contains no paragraphs and no reading order.

What it stores is a pile of drawing instructions: on which page, at which coordinates, at what size, draw this run of text. Paragraphs, heading levels, and reading order are all inferred by you from coordinates and font sizes. What a real parsing library (pdfplumber in Python, pdfjs in JavaScript) hands you is exactly such a list of records. Three mountains follow.

The first mountain is multi-column reading order. Layout software often writes the content stream in bands from top to bottom by y value. That is fine for a single-column document; for two columns it comes out as left column line one, right column line one, left column line two, right column line two, alternating. Copy that order directly and the text is two columns interleaved — and each sentence alone reads fine, with the incoherence appearing only when read continuously. The fix is rebuilding the reading order yourself: sort each page's text blocks by left edge, find a gap wide enough to be a column boundary, then reorder by column, then y descending, then x ascending.

The second mountain is tables. A table in a PDF is a pile of aligned text blocks, with the rules drawn as graphics unrelated to the text. Extracted, a whole table may collapse into one line or each cell may become its own paragraph. The fix has two layers: where positions are available, cluster by x coordinate to restore columns; where they are not, at least recognize that it collapsed and mark it low-confidence and excluded from retrieval rather than letting it masquerade as body text.

The third mountain is scans. Some PDFs are one image per page with not a character in the text layer. The only road then is optical character recognition (OCR), reading the image as text. Two costs: either installing a system-level dependency or paying per page for a cloud service; and it will certainly make mistakes. In Chinese those mistakes are mostly visually similar characters, and in English they are letter confusions — rn read as m, l read as 1, O read as 0. Those errors propagate down the chain: tokenization splits wrongly, a term is missing from the inverted index, retrieval fails to hit, and the material the model sees carries typos. The mitigation is accepting it and compensating: keep a link to the original image for human review, let D9's vector path cover the keyword path's misses, and mark low-confidence pages separately.

HTML and Markdown: easy to parse is not easy to use

Compared with PDF, HTML and Markdown are a gift — they carry their own structure. A heading is a heading tag, a paragraph is a paragraph tag, a list is a list tag, with nothing to guess from coordinates. Markdown needs no parsing library at all; one line-by-line scan does it.

But easy to parse is not easy to use, and their pitfall is at the other end: over half of a web page's content does not belong in the body. Navigation bars, a sidebar's related documents, a footer's registration number, analytics scripts embedded in the page — all identical on every page. Keeping them adds the same passage to every document in the store, lowering discrimination and letting users retrieve content with no information in it. So the first step of HTML parsing is not extraction but wholesale deletion: scripts, styles, navigation, sidebars, footers, all swept out by selector.

Three things are easily dropped along the way and are a shame to lose:

• Code blocks have meaningful spaces and newlines, and running them through the ordinary paragraph's whitespace-collapsing rule kneads them into one line where the reader can no longer see the indentation levels.
• Footnotes leave only a superscript number in the body with the real content at the page's foot. Extract without reattaching it to the reference point and the user sees a lonely bracketed number.
• Relative links such as ./doc-002.md are genuine metadata telling you which document this one points at. Dropping it takes one line of code and recovering it means reparsing the original. Today's lab inlines it into the text — writing "the task board (link: ./doc-001.md)" — adding no field and losing nothing.

Markdown has far fewer pitfalls, mainly two: a blank line inside a code fence must not act as a paragraph separator (or a shell snippet gets cut into three unrelated chunks), and a heading must not become a node of its own — it is too short and worthless retrieved alone; the right approach makes it the heading path of the following nodes, which is the next section's subject.

One intermediate representation: every source becomes the same stream of nodes

Three parsers, three output formats, and eleven days of writing handling logic for each? Of course not. Parsing's real product is converging every source onto one data structure: a stream of nodes with structure and provenance. From there on, D4's chunking, D6's citations, and D13's permission filtering know only this structure and no longer care whether a document was a PDF or a web page.

This course fixes that structure at eight fields, not to be renamed across fourteen days:

import { createHash } from 'node:crypto'
 
// The course-wide parse node. One node = one parse unit (a paragraph, a list, a table)
export function makeNode(meta, index, text, headingPath, page) {
  return {
    docId: meta.docId,
    chunkId: `${meta.docId}#c${String(index + 1).padStart(2, '0')}`,
    text,
    headingPath: [...headingPath],
    page, // only PDF sources have a page; every other source is null, never 0 and never blank
    department: meta.department, // the permission label, used directly by D13's access control
    updatedAt: meta.updatedAt,
    contentHash: meta.contentHash, // one fingerprint shared by the whole document
  }
}
 
// Normalize newlines before hashing: the same file uploaded from two machines differs in bytes
// and matches in content, and without normalization every sync judges it changed and recomputes the store
export function contentHash(raw) {
  const normalized = raw.replace(/\r\n/g, '\n').trim()
  return createHash('sha256').update(normalized, 'utf8').digest('hex').slice(0, 16)
}

import hashlib
from dataclasses import dataclass, field
 
 
@dataclass
class ParsedNode:
    """The course-wide parse node. One node = one parse unit (a paragraph, a list, a table)"""
 
    doc_id: str
    chunk_id: str
    text: str
    heading_path: list[str] = field(default_factory=list)
    page: int | None = None  # only PDF sources have a page; every other source is None
    department: str = ""  # the permission label, used directly by D13's access control
    updated_at: str = ""
    content_hash: str = ""  # one fingerprint shared by the whole document
 
 
def make_chunk_id(doc_id: str, index: int) -> str:
    return f"{doc_id}#c{index + 1:02d}"
 
 
def content_hash(raw: str) -> str:
    """Normalize newlines before hashing: the same file from two machines differs in bytes and
    matches in content, and without normalization every sync judges it changed and recomputes"""
    normalized = raw.replace("\r\n", "\n").strip()
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()[:16]

Among the eight, headingPath is the most often written wrongly. It is not this node's previous heading but the complete path down from the level-one heading, such as "Plans and Quotas / Notes." Maintaining it needs only a stack: on reading a level-n heading, truncate the stack to n minus 1 and then append. That truncation is the key — retreating from level three to level two must discard the third level, or the path staggers and a reader following the citation back lands in a different section.

export function pushHeading(stack, level, title) {
  const next = stack.slice(0, level - 1) // retreating a level must discard the deeper headings
  while (next.length < level - 1) next.push('') // level one jumping to level three; pad to avoid staggering
  next.push(title)
  return next
}
 
// On reading "## Swimlanes": pushHeading(['Task Board Guide'], 2, 'Swimlanes')
// gives ['Task Board Guide', 'Swimlanes'], and every later node carries that path

def push_heading(stack: list[str], level: int, title: str) -> list[str]:
    nxt = stack[: level - 1]  # retreating a level must discard the deeper headings
    nxt += [""] * (level - 1 - len(nxt))  # level one jumping to level three; pad to avoid staggering
    nxt.append(title)
    return nxt
 
 
# On reading "## Swimlanes": push_heading(["Task Board Guide"], 2, "Swimlanes")
# gives ["Task Board Guide", "Swimlanes"], and every later node carries that path

As for a multi-column PDF's reading order, the code to rebuild it is shorter than expected: find the gap, split the columns, go top to bottom within a column.

// Sort every text block's left edge on a page; the largest gap is the column boundary
export function detectColumnBoundary(runs) {
  const xs = [...new Set(runs.map((r) => Math.round(r.x)))].sort((a, b) => a - b)
  let best = { gap: 0, left: 0, right: 0 }
  for (let i = 1; i < xs.length; i += 1) {
    const gap = xs[i] - xs[i - 1]
    if (gap > best.gap) best = { gap, left: xs[i - 1], right: xs[i] }
  }
  if (best.gap < 60) return null // the gap is not wide enough; judged single-column
  const boundary = (best.left + best.right) / 2
  const ratio = runs.filter((r) => r.x >= boundary).length / runs.length
  // Both sides need a decent volume of text, or that "gap" is most likely just an indent
  return ratio > 0.15 && ratio < 0.85 ? boundary : null
}
 
export function sortRuns(runs) {
  const boundary = detectColumnBoundary(runs)
  const col = (r) => (boundary === null || r.x < boundary ? 0 : 1)
  return [...runs].sort((a, b) => col(a) - col(b) || b.y - a.y || a.x - b.x)
}

def detect_column_boundary(runs: list[dict]) -> float | None:
    """Sort every text block's left edge on a page; the largest gap is the column boundary"""
    xs = sorted({round(r["x"]) for r in runs})
    gaps = [(xs[i] - xs[i - 1], xs[i - 1], xs[i]) for i in range(1, len(xs))]
    if not gaps:
        return None
    gap, left, right = max(gaps)
    if gap < 60:  # the gap is not wide enough; judged single-column
        return None
    boundary = (left + right) / 2
    ratio = sum(1 for r in runs if r["x"] >= boundary) / len(runs)
    # Both sides need a decent volume of text, or that "gap" is most likely just an indent
    return boundary if 0.15 < ratio < 0.85 else None
 
 
def sort_runs(runs: list[dict]) -> list[dict]:
    boundary = detect_column_boundary(runs)
    def col(r: dict) -> int:
        return 0 if boundary is None or r["x"] < boundary else 1
    return sorted(runs, key=lambda r: (col(r), -r["y"], r["x"]))

The boundary of cleaning: what to delete, and what can never be recovered

Cleaning has a trap: deleting feels great and everything looks tidier afterwards. There is one criterion —

Could it be recovered from the original afterwards? If yes, delete boldly; if no, keep it for now.

By that criterion, cleaning actions fall into two classes.

Safe to delete: scripts and styles, navigation bars and footers, page headers repeated on every page, purely decorative whitespace and rules, zero-width characters and invisible control codes. What they share is having no information in the original either, so deleting them any number of times loses nothing. Headers and footers have an extra advantage: their position on each page is fixed, so cutting a band off the top and bottom by y value works — but remember to print how many were removed and confirm no body text was hit; in today's lab a two-page single-column PDF should remove exactly four.

Unrecoverable once deleted: page numbers (D6's citations point back to a page), heading levels (lost, and a node becomes an isolated stretch of text with no context), relative links and source addresses, a table's row-column relationships, and the update time. Once any of those is lost at parse time, the only remedy is reparsing the original — and three years later you most likely no longer have the original, only an index.

One class sits between the two: newlines and whitespace. Collapsing whitespace is right in body text and a disaster in a code block. So cleaning rules cannot be applied globally and must branch by node type — one of the reasons the unified intermediate representation records a block type.

Metadata is the foundation of every later feature

Now revisit those eight fields and say who each serves — this section is today's most important, because every later day comes back to draw on one of them.

• docId and chunkId: a chunkId shaped like doc-005#c07 is the anchor of the whole course's citation system. When D6 has the model mark a source, that id is what it marks; when citation validation checks back against the original, that is what it checks. Without a stable id there is no verifiable citation, and an answer's credibility falls straight back to whatever the model says.
• text: the cleaned body, consumed by retrieval and generation.
• headingPath: one line summarizing the node's position in the whole document. It has two uses: letting a citation tell the user which section a sentence came from, and giving D4's structure-based chunking natural semantic boundaries in the heading levels — provided it was not dropped today.
• page: only PDF sources have it. D6's citations need page precision so a user opening the original lands in the right place.
• department: the permission label. D13 on access control proves one thing — permission filtering must sink into the retrieval query, since filtering at generation time means the content already leaked. This field is that filter condition, and not stamping it today means reparsing the whole store then.
• updatedAt: D1's lab already showed its use: when two documents' conclusions clash, the prompt requires the model to list both with their update dates and let the user judge.
• contentHash: the fingerprint of the whole document's content. D13's incremental sync uses it to decide whether a document needs recomputing — reparse, rechunk, and re-embed only what changed. Without it every sync is a full rebuild, and recomputing a three-thousand-document knowledge base daily makes the embedding bill alone worth looking at.

// A collapsed table: one table parsed into a shape that does not line up
export function detectCollapsedTable(text) {
  const lines = text.split('\n').filter((l) => l.trim())
  const tableLines = lines.filter((l) => (l.match(/\|/g) ?? []).length >= 2)
  if (tableLines.length === 0) return false
  if (tableLines.length === 1) return true // the whole table collapsed into one line
  const widths = new Set(tableLines.map((l) => l.split('|').length))
  return widths.size > 1 // rows with disagreeing column counts have crossed
}
 
// The empty text ratio: almost no characters extracted usually means a scan with no text layer
export function emptyRatio(nodes) {
  if (nodes.length === 0) return 1
  return nodes.filter((n) => n.text.trim().length < 4).length / nodes.length
}

def detect_collapsed_table(text: str) -> bool:
    """A collapsed table: one table parsed into a shape that does not line up"""
    lines = [l for l in text.split("\n") if l.strip()]
    table_lines = [l for l in lines if l.count("|") >= 2]
    if not table_lines:
        return False
    if len(table_lines) == 1:
        return True  # the whole table collapsed into one line
    widths = {len(l.split("|")) for l in table_lines}
    return len(widths) > 1  # rows with disagreeing column counts have crossed
 
 
def empty_ratio(nodes: list) -> float:
    """The empty text ratio: almost no characters extracted usually means a scan with no text layer"""
    if not nodes:
        return 1.0
    return sum(1 for n in nodes if len(n.text.strip()) < 4) / len(nodes)

One last note on something today does not do: chunking. A node in those eight fields is a parse unit — a paragraph, a list, a table — with boundaries set by the document's own structure. It is not necessarily the chunk used for retrieval: a paragraph may be too short and a table too long. How nodes are recombined into retrieval chunks is tomorrow's whole subject, and tomorrow proves that decision must rest on evaluation rather than intuition. Today does one thing only: turn documents into a stream of nodes with structure, provenance, and metadata, losing not one field.

Source Reading

Hands-On Lab

The lab's PDF and web page samples are not downloaded; they are manufactured from D1's corpus by a piece of plain JavaScript: a minimal PDF byte stream written by hand, with text plus page numbers, coordinates, and font sizes in the content stream, deliberately laid out in two columns with that already-crossed table pressed in. That has two benefits — clean provenance, and offline reproducibility, so it runs on a plane. The scan is likewise simulated: no OCR engine is installed, and a noisy plain-text version is manufactured by a rule of visually-similar substitution plus structure flattening, with the errors injected deterministically so every run matches.

starter/ has four exercise points cut out: the heading stack, two-column reading order, the collapsed table assertion, and the content fingerprint. Run as-is, every node shows "(no heading path)", the two-column PDF's disorder rate will not come down, and that broken table sails through — the four exercises each map to a real class of failure, and only finishing them turns everything green.

1. Run the starter once and read all seven sections of output, noting which assertions are red and on which document each one is.
2. Complete the heading stack, rerun, and watch the sample line go from "(no heading path)" to a full path such as "Task Board Guide / Swimlanes".
3. Open the output for the generated two-column PDF sample, reorder the reading order by column, and watch the disorder rate fall from 0.33 to 0.00 while the "first four paragraphs in order" line becomes coherent.
4. Implement the collapsed table assertion, rerun, and see the table missing a column separator flagged red on both the Markdown original and the PDF path.
5. Complete the content fingerprint, confirm the three sources' fingerprints all differ, then append one sentence to one of them and recompute, watching the fingerprint change — the entire basis of day 13's incremental sync.

Interview Questions

Today's 4 questions are in the question bank below, weighted toward how parsing quality transmits into retrieval, metadata design, and detecting and blocking dirty data. Expand each one and read the analysis before the key points — practicing the derivation beats memorizing bullets. The high-frequency labels for the domestic and overseas markets are there so you can pick by target market.

Checklist and Tomorrow

• Name three typical PDF parsing failures — multi-column reordering, tables collapsing into one line, headers and footers leaking into the body — and give a fix for each
• Design a document metadata structure that gives later citation labeling, permission filtering, and incremental updates something to rely on
• Write a parsing pipeline that normalizes documents from multiple sources into one intermediate representation, and run an assertable quality check on the parsed result
• Say for each of the eight fields which feature of which later day it serves, especially the permission label and the content fingerprint
• State the "could it be recovered from the original" cleaning criterion, with two examples each of what to delete and what to keep
• All 5 acceptance criteria of the lab pass, with out/nodes.json generated
• Answer at least 3 of the 4 interview questions without looking at the key points

Tomorrow (D4) we recombine today's stream of nodes into retrieval chunks — chunking — implementing five approaches: fixed length, recursive, structure-based, parent-child, and semantic. The order is deliberate: many chunking options exist only because parsing preserved the structure, "cut by heading level" being exactly one whose precondition is today's headingPath. More importantly, tomorrow establishes a rule that never moves again: how large a chunk should be cannot rest on intuition and can only rest on running an evaluation and reading the numbers.