Getting Documents In: Parsing PDF and HTML, Tables and Scans, Cleaning Rules, and Metadata You Must Keep
Retrieval quality's ceiling is parsing quality; today go all the way through the parsing pitfalls of PDF, HTML, and Markdown sources, handle tables and scanned pages, settle on a set of cleaning rules, and lock down the metadata — heading level, page number, source address — you'll need later for citations and filtering.
Retrieval quality's ceiling is parsing quality; today go all the way through the parsing pitfalls of PDF, HTML, and Markdown sources, handle tables and scanned pages, settle on a set of cleaning rules, and lock down the metadata — heading level, page number, source address — you'll need later for citations and filtering.
Interview questions
The text extracted from a PDF comes out in the wrong order. How do you diagnose and fix it?一份 PDF 解析出来的文字顺序是乱的,你会怎么排查和修复?
Common in ChinaCommon overseasIntermediate#pdf-parsing#ingestion#data-quality🔒 Unlock this chapter to see the answer keys.
Which metadata should a document parsing stage preserve, and which downstream feature breaks if you drop each one?文档解析阶段应该保留哪些元数据?少了其中某一项会在哪个环节出问题?
Common in ChinaCommon overseasIntermediate#metadata#ingestion#access-control🔒 Unlock this chapter to see the answer keys.
OCR output from scanned documents carries a non-trivial error rate. How does that noise propagate into retrieval and generation, and how do you mitigate it?扫描件走光学字符识别之后错字率不低,这些噪声会怎样影响检索和生成?怎么缓解?
Common in ChinaCommon overseasDeep dive#ocr#data-quality#hybrid-search🔒 Unlock this chapter to see the answer keys.
Why is parsing quality the ceiling on retrieval quality? Walk through one concrete chain of propagation.为什么说解析质量决定了检索质量的上限?举一个具体的传导链条。
Common in ChinaCommon overseasBasic#ingestion#data-quality#failure-analysis🔒 Unlock this chapter to see the answer keys.
