Key takeaways

The original is immutable; the index is derived. Keep the raw blob, each transform, and search representations apart. A new parser must be able to replay the corpus without re-fetching SharePoint.

Parsing is a measured product, not a one-off utility. Parser version, OCR confidence, and flags such as “table / footer / scan” belong in the document manifest. A silent parse failure pollutes both lexical search and vectors.

A table cell without its header is not evidence. A value torn from row and column labels yields a confident wrong answer. Layout beats “extracted text.”

Version and deletion are part of the freshness contract. One source_id with revision history, a validity window, and a cascade onto chunks, cache, and citations. “Re-upload the folder” creates twins.

Quarantine is more expensive than silence and cheaper than an incident. A low-confidence scan or a broken structure must not enter the live index “just in case.”

What a document ingest pipeline is

Ingest starts before embeddings. If this step permutes the columns of an estimate, no hybrid retriever will save you: both channels will honestly rank the false text. The organizational layer — what counts as a source, who owns freshness — lives in RAG data preparation and enterprise RAG architecture. This article is the engineering contour: idempotency, parsers, OCR, tables, versions.

The practical boundary: any component that sees document text after parse — chunking, index, eval, debug export — consumes the canon, not “whatever the PDF produced tonight.” If you cannot reproduce the canon from the original plus parser version, you do not have ingest. You have a one-time migration.

What it is not

It is not “pick Unstructured or Docling.” The library is a stage detail. The contract is inputs, artifacts, failure, version.

It is not chunking. Fragment boundaries depend on structure ingest must preserve: headings, lists, cells, figure captions. If the parser flattened the document into a sheet, chunking experiments are measuring a spoiled input.

It is not access control. Rights are copied onto the document at intake and updated by events, but the decision “may we search this” stays with filtering before retrieval. The loader must not write text into a shared index that nobody can filter.

Why the parser outranks the embedding

Teams swap the embedding model when users complain about “stupid answers.” The parse is often guilty.

A typical chain: a two-column PDF is read left-to-right across both columns; a negation (“does not apply to contractors”) is torn from its condition; a repeating footer “confidential, p. 14” becomes a frequent token; in a table “rate / 12%” the 12% is indexed without the header. The embedding honestly treats that as meaning. BM25 honestly promotes the footer on the word “confidential.”

A second failure class is missing text. A scan with no OCR yields an empty canon. The system indexes the file name and path. Content search is silent; name search lies. A third class is mixed: half the pages have a text layer, half are images. A naive parser extracts only the text layer and silently drops scanned annexes.

Immutable original and canon

A working scheme has three layers.

Layer 0 — original. File bytes, MIME, size, checksum, id in the source system, source_updated_at. This object is never overwritten. If the source sends a new file with the same name, that is a new version, not an overwrite.

Layer 1 — canon. A structured representation: section tree, blocks (paragraph, list, table, figure, footer), language, page geometry if present, OCR confidence, parser version, warnings. JSON or Markdown with explicit table markup is fine as long as it is stable and diffable. Canon can be recomputed; the original cannot.

Layer 2 — search representations. Chunks, lexical fields, vectors. You can burn them and rebuild from canon. If you can “reindex from canon,” you are already ahead of most demos.

Minimum document manifest:

Reprocessing the same content_hash with the same parser_version must not create a second record. A parser bump creates a new canon of the same original version and archives the old canon. That is how you compare parsers on one corpus.

Text, layout, and scans

“PDF” is not one format. It is three different intakes.

Digital PDF with a text layer

There are glyphs, fonts, coordinates. The job is to recover reading order and blocks, not to concatenate extractText(). Multi-column layout, callouts, and footnotes need layout analysis. Libraries such as PyMuPDF, pdfminer, Docling, and commercial pipelines differ here, not in “can open the file.”

Test on your layouts: a two-column policy, a form with a side nav, a product datasheet with a frame. There is no universal winner. There is a set of gold pages and a metric: block order matches human reading, footnotes did not weave into the paragraph, footers are separated.

Scans and image-only PDFs

There is no text layer, or it is empty junk. You need OCR. Persist confidence per page and per block. Low confidence means quarantine or a retry with another recognition model and language — not silent indexing. Set document language explicitly: autodetection on a letterhead with Latin in the header and Cyrillic in the body often picks the wrong engine.

Do not mix “confident text layer” and “doubtful OCR” in one canon without a flag. Otherwise retrieval eval cannot tell a bad embedding from a bad “з” instead of “3.”

Hybrids

Many corporate PDFs are text plus embedded scans of annexes. The pipeline should walk pages: dense enough text layer → layout parse; otherwise OCR. Attachments (nested PDFs, TIFF) are separate documents with a parent_source_id, not “another page of the same file,” if they have their own numbering and marking.

Office files and mail

DOCX and Excel sheets often look “easier than PDF,” and teams skip the contour. Word revisions, embedded objects, hidden sheets, and merged cells break canon the same way a two-column policy does. For tables, the .xlsx source is usually more honest than a “print to PDF” report: the grid already exists; you do not have to guess it from coordinates. The printed PDF still matters when that is the legally signed artifact — then the PDF is the original, and a separately emailed spreadsheet is a related source, not a parser’s guess.

An email is not one document. Subject, body, reply chain, and each attachment have different markings and different value as evidence. Mail canon stores headers, participants, and links to child attachments; attachments follow the same pipeline as ECM files. Otherwise a search for “that email with the contract” retrieves the manager’s signature block.

Tables, footers, and false facts

Tables are the main source of confidently wrong answers in production RAG.

A cell without header context is useless. “12” may be a rate, a quantity, a year, or an error code. Table canon must carry column headers, a row header if present, merged cells, table number, and caption. Retrieval often wants two views: structured “column=value” rows for exact questions and serialized Markdown/CSV for the model. Both are built from one parsed grid, not from “nearby text.”

Broken cell order (reading by columns instead of rows, or the reverse) yields a plausible paragraph. That is worse than an empty result: the model cites “evidence.” On gold tables measure key→value match, not text BLEU.

Running headers and footers must be classified and stripped from the body. Otherwise BM25 falls in love with “Romashka LLC, confidential.” Figure captions, conversely, must not be dropped: “what is on diagram 3” searches the caption, not pixels. The “figure — caption — number” bond lives in the canon before chunking.

Footnotes: either bind them to the paragraph with a marker, or keep a numbered block. Gluing a footnote to the next paragraph changes legal wording.

Versions, deletes, idempotency

Sources rarely hand you “here is the current file.” They hand you events, folders, and colliding names.

Idempotent intake

Key (source_id, content_hash) or (source_id, source_version). A replayed event must not mint new index points. A rename without a hash change is the same content — update the path in the manifest. A hash change is a new version: the old one is marked superseded_by, valid_to closes, and fragments of the old revision are either removed from current search or kept for “as-of” queries.

Legal and finance contours almost always need historical search. Do not delete the previous canon while retention policy allows it. Remove it from the active index.

Deletion

A delete event must cascade: canon status deleted, chunks, vectors, lexical index, answer cache, citations stored in sessions. “The file left the folder, chat still cites it” is an ingest defect, not a generation defect. For permission revocation see ACL before retrieval: a different trigger, the same cascade onto derivatives.

Version conflicts

Two connectors bring one source_id with different hashes in the same minute. You need a rule: the system of record wins (ECM, not the mailbox), or manual quarantine. Silently taking “last ingested” is a lottery.

Quarantine and parse quality

Quarantine is a queue of documents that must not enter live search without a decision. Causes: OCR below threshold, zero extracted blocks on a non-empty file, a table with impossible geometry, an unsupported PDF cipher, a language outside policy, a malware scan, a parser exception.

Each quarantine record: original, error, parser version, source owner, deadline. Silently skipping “we’ll parse it later” means the user believes the document is in the system. Prefer an explicit source-card status: “accepted, not indexed.”

Parser eval is its own contour, not to be confused with the question golden set. A page set: two-column prose, a mid-quality scan, a table with merged cells, a footer, a mixed PDF. Metrics: block order, table key F1, share of pages in quarantine, regression on a parser_version bump. A new parser runs this set first, then replays canon, then rebuilds the index.

Do not compare parsers “by eye on one contract.” Compare them on a fixed corpus snapshot and watch which retrieval slices moved: identifiers, tables, scans.

Events, batches, and the freshness contract

Connectors differ: SharePoint and disks emit deltas, mail emits messages with attachments, Git emits commits, ERP emits attachments on a card, a shared folder emits nothing. A single “nightly script over all shares” hides lost events.

A practical contour: a change stream (webhook, change log, queue) plus periodic hash reconcile. Reconcile catches what the stream dropped. The stream gives freshness. Without reconcile you will one day find you have been indexing a ghost of a deleted file for three months.

The freshness contract is already named in the pillar: source_updated_at, ingested_at, indexed_at, deletion, valid_from / valid_to. Ingest owns the first two and “canon ready for index.” If ingested_at − source_updated_at stably exceeds the source SLO, the LLM is not the problem.

Observability: documents accepted, documents in quarantine, ocr_confidence distribution, share of tables with a recovered header, reconcile lag, connector errors by source. Without those series, “the index updates” is a slogan.

Common mistakes and a four-week plan

Common mistakes

One extractText call on the whole PDF. You lose layout, tables, and half the scans.

Overwriting the original. You cannot prove the canon matches the file counsel saw.

Indexing from quarantine “for coverage.” Coverage with garbage is worse than a hole: the model cites mush.

Version = file name. policy_final_v3_new(2).pdf is not a version_id.

Shipping a parser bump with an embedding bump. You cannot attribute the regression.

Rights only in the source, not on the manifest. After export the ACL link breaks; see multi-tenancy.

Evaluating only chat answers. The parser looks green until you slice table queries.

Four weeks to controlled intake

Week 1 — inventory and layers. Registry of sources, formats, volumes. Introduce an original layer and a ban on overwrite. Count how many files currently enter the index without a hash and parser version — that is debt.

Week 2 — canon and quarantine. One canon format, a quarantine queue, media_class and OCR flags. A gold set of 20–40 pages from your layouts, not demo PDFs from the internet.

Week 3 — tables and hybrid PDFs. A separate table representation, footer stripping, per-page “text or OCR.” Run the gold set, freeze a baseline.

Week 4 — versions and reconcile. Idempotent key, delete cascade, nightly reconcile of one source. Replay canon with a new parser without re-fetch. If that is impossible, the contour is not ready.

If intake needs to become an observable service with contracts rather than an analyst script, the AI implementation service is that bundle: sources, parse, evaluation.

FAQ

Further reading

• Production RAG engineering — the full pipeline; this article closes intake.
• RAG data preparation — source policy and corpus quality.
• Enterprise RAG architecture — a platform around many sources.
• Chunking experiments — cuts after an honest canon.
• ACL and multi-tenancy — rights copied at intake, filter before retrieval.
• Golden dataset for RAG eval — where to add table and scan slices.
• Prompt injection in production — every accepted file as an input channel.

Conclusion

Production RAG stops lying with tables the moment ingest becomes its own product: an immutable original, a reproducible canon, quarantine instead of silence, versions instead of final_final.pdf. Embeddings and rerankers run on top of that contour, not instead of it.

This week, take one verifiable action: forbid overwriting originals and write content_hash plus parser_version on every accepted file. Then quarantine everything currently indexed with no text layer and no OCR. The rest is thickening the same border: layout, tables, reconcile, parser replay without a re-fetch.

Document parsing is boring. It is also what separates corporate search from a folder of PDFs with a polite model on top.