Contents
Introduction
After architecture, model, and infrastructure have been selected, the decisive question is what data the enterprise AI system can use. In practice, data preparation often matters more to answer quality than a newer model release. A model cannot correct duplicate files, obsolete policies, or disorderly archives.
Why it matters
RAG is only as reliable as its sources. Contradictions and outdated documents become contradictions and outdated guidance in generated answers. Data preparation is therefore an engineering process with clear rules, accountable owners, and measurable outcomes.
Core explanation
A preparation pipeline inventories sources, cleans and normalises documents, removes duplicates, extracts metadata, splits content into meaningful chunks, creates embeddings, indexes it, and keeps the index current.
flowchart LR
sources[Data sources] --> cleaning[Cleaning]
cleaning --> normalize[Normalisation]
normalize --> chunks[Meaningful chunks]
chunks --> embeddings[Embeddings]
embeddings --> index[Vector index]
index --> rag[RAG]
There is no universal chunk size. Large chunks reduce retrieval precision; very small ones remove the context needed to answer safely. Choose the strategy for the document structure and user scenarios, then test it against real questions.
Metadata is as valuable as text. Author, approval date, version, business unit, category, validity period, and confidentiality level improve retrieval and support access controls.
Security
| Requirement | Practical effect |
|---|---|
| Document classification | Restricted material is not indexed without approval |
| Currency checks | Search uses the current version |
| Source ownership | Data quality and change responsibility are clear |
| Update logging | Data lineage remains traceable |
| Role-aware filtering | Retrieval can enforce each user's permissions |
Access rights must travel through the data pipeline; they cannot be reliably bolted onto an index afterwards.
Enterprise example
An industrial group collects documents from ERP, document management, file archives, and an internal portal. A single pipeline removes technical noise, attaches metadata, creates chunks, and applies quality checks. The resulting knowledge base is consistent even though the source systems are not.
Industrial safety example
Safety experts need regulations, internal methods, inspection reports, and assessment archives. Older regulations remain available for historical analysis but are removed from routine retrieval. This reduces the chance that a report cites a rule that is no longer valid.
AI can assemble evidence from authorised sources, while the expert checks the basis of the conclusion and remains accountable for the decision.
Common mistakes
- Indexing documents before cleaning them.
- Ignoring versions and validity periods.
- Applying the same chunking rule to every data type.
- Neglecting metadata and source ownership.
- Failing to refresh the index.
Practical conclusions
Before launch, define a source catalogue, cleaning rules, a metadata model, a chunking strategy, an update schedule, and a quality-control procedure. These turn archives into a managed knowledge base instead of another file repository.
Key takeaways
- Data quality matters more than document volume.
- Preparation needs its own process and ownership.
- Metadata improves retrieval and enables security controls.
- Version management is essential in an enterprise.
- A knowledge base must be continuously maintained.