Key takeaways

Treat RAG as a search product, not an LLM feature. Retrieval quality, document freshness, access control, and source provenance usually determine the ceiling on answer quality. Generation cannot recover evidence that retrieval never supplied.

There is no universal chunk size, top-K, similarity threshold, or RRF constant. These values depend on document structure, query distribution, embedding model, latency budget, and the cost of a wrong answer. Tune them against a representative evaluation set.

Hybrid retrieval is a strong default, not a law. Dense vectors are useful for paraphrases and conceptual similarity; lexical methods such as BM25 remain excellent for exact identifiers, names, error codes, and rare terms. Fusion and reranking can combine their strengths, but every additional stage adds latency and operational complexity.

Security filtering belongs before model context. Metadata filters and authorization checks must constrain retrieval before text reaches a reranker or language model. Asking the model to ignore unauthorized passages is not access control.

Evaluate retrieval and generation separately. A fluent wrong answer may be a retrieval failure, a context-assembly failure, or a generation failure. Without stage-level metrics and traces, teams optimize whichever component is easiest to change.

Why demo RAG breaks in production

A demo corpus is small, clean, static, and usually selected by the same person who writes the demo questions. Production reverses every assumption. The corpus contains duplicated policies, scanned PDFs, stale wiki pages, spreadsheets, tickets, code, and contradictory versions. Queries are short, misspelled, conversational, multilingual, or built around internal vocabulary.

Several failure modes then interact:

• a nightly export misses a policy update;
• a parser drops a table header and turns cells into meaningless text;
• fixed-size splitting separates a condition from its exception;
• semantic search misses an exact incident ID;
• dense and lexical results contain near-duplicates;
• the reranker promotes an old but well-written document;
• context truncation removes the sentence that changes the answer;
• the model answers despite weak evidence;
• a cache serves a response produced before an ACL change.

The visible symptom is simply “RAG hallucinated.” That label is too broad to be actionable. A production design must expose where evidence disappeared or became corrupted. For a broader treatment of boundaries and governance, see enterprise RAG architecture. The central lesson is that answer quality is an end-to-end property, while debugging must remain stage-specific.

A reference architecture with explicit contracts

A durable RAG platform separates control flow from data flow. One practical decomposition has five planes.

Ingestion and normalization

Connectors read systems of record through snapshots, change-data capture, webhooks, or queues. Parsers extract text and structure. Normalization assigns stable document IDs, source timestamps, ownership, language, ACL metadata, and content hashes. A rejected-document queue captures parser failures instead of silently omitting them.

Indexing

An indexing worker applies a versioned chunking policy, generates embeddings with a pinned model, and writes lexical and vector representations. It also stores a document registry linking every chunk to its source revision. Index creation should be idempotent: replaying the same source event must not create duplicate chunks.

Query and retrieval

The query service authenticates the principal, normalizes or rewrites the question, applies tenant and ACL constraints, retrieves candidates, fuses result lists, and optionally reranks them. Its output is not an answer. It is a typed evidence package with scores, provenance, versions, and filter decisions.

Context and generation

The generation service assembles a bounded context, removes redundant passages, preserves citations, and asks the model to answer or abstain. The response should link claims to source spans, not merely append a list of documents.

Evaluation and operations

Offline suites compare pipeline versions. Online telemetry captures latency, cost, retrieval results, citations, abstentions, and user feedback. Deployment gates, shadow traffic, canaries, and rollback controls connect evaluation to releases. This plane turns “we changed the chunk size” from an experiment into a controlled migration.

Keep these boundaries even if one process implements several stages initially. Clear schemas make it possible to replace an embedding provider, vector engine, reranker, or generator without rebuilding the entire application.

Start with a source inventory and freshness contract

Teams often begin by selecting a vector database. The better first artifact is a source inventory. For every source, record:

• system owner and technical connector;
• authoritative status and precedence;
• supported document types and languages;
• expected update frequency and maximum acceptable staleness;
• deletion and retention rules;
• identity and ACL model;
• parser coverage, including tables, images, and attachments;
• volume, growth, and backfill strategy;
• data classification and residency constraints.

Define authority before resolving contradictions

Two documents can both be semantically relevant and disagree. Retrieval scores cannot decide which one is authoritative. Add explicit precedence: an approved policy may outrank a wiki page; a current product manual may outrank an archived release. Include effective dates and lifecycle states such as draft, active, superseded, and deleted.

Measure freshness as a pipeline

“Indexed at” is not enough. Record source modification time, connector observation time, normalization time, and index availability time. Their differences distinguish source lag, connector lag, queue delay, and indexing delay. Set freshness objectives by source: a security revocation may require near-real-time propagation, while a monthly research archive may not.

Use content hashes to avoid unnecessary work, tombstones to propagate deletions, and reconciliation jobs to catch missed events. More detailed preparation patterns are covered in RAG data preparation.

Chunking is a document-modeling decision

Chunking controls the unit a retriever can find and the unit a model can cite. If chunks are too broad, relevant sentences compete with unrelated text and consume the context budget. If they are too narrow, they lose headings, qualifiers, and relationships. A token count alone cannot solve this tradeoff.

Layout-aware chunking

Layout-aware parsing uses headings, paragraphs, lists, table boundaries, code blocks, page regions, and captions. It should preserve a hierarchy such as document → section → subsection → block. For manuals, contracts, and reports, this usually produces more meaningful evidence than splitting every fixed number of tokens.

Tables need special handling. Repeating column headers in each serialized row group is often more useful than flattening the entire table. For code, split at symbol or syntax boundaries and attach module, class, and signature metadata. For slide decks, keep titles and speaker notes associated with visible content.

Semantic chunking

Semantic chunking detects topic transitions using sentence similarity or a model. It can help with long prose that has poor headings, but it costs more and can be unstable across model versions. A semantically coherent chunk may also become too large for precise retrieval. Keep deterministic limits and store the policy version.

Parent-child retrieval

Parent-child indexing embeds smaller child passages for precise matching while returning a larger parent section for context. This separates retrieval granularity from generation granularity. It works well when a single paragraph identifies the topic but surrounding paragraphs contain definitions and exceptions.

The danger is context inflation. If five matching children expand into five large parents, the system can exhaust its token budget and duplicate content. Group children by parent, cap expansion, and evaluate both retrieval and final context.

Late chunking

Late chunking creates token representations with awareness of a longer document context and then pools them into chunk embeddings. It can preserve references that ordinary independent chunks lose. It is not a universal replacement for structure-aware parsing: model context limits, compute cost, unsupported formats, and embedding infrastructure still matter.

Why no universal chunk size exists

A support knowledge base, legal corpus, source-code index, and biomedical paper collection have different semantic units. Even within one corpus, a glossary entry and a troubleshooting procedure should not share identical splitting rules. Start with structure, define minimum and maximum bounds, and test several policies. Report results by query slice rather than selecting a size from a blog post.

Version embeddings and index artifacts

An embedding vector is meaningful only in the space produced by its model and preprocessing. Changing model, dimensions, normalization, tokenizer, input prefix, or truncation policy can invalidate comparisons with existing vectors.

Create an embedding manifest containing at least:

• provider and exact model identifier;
• model revision when available;
• dimensions and distance function;
• normalization and input-template rules;
• maximum input and truncation behavior;
• chunking and parser versions;
• creation time and code revision.

Store embedding_version on every record. Do not overwrite an index in place during an upgrade. Build a parallel version, run offline evaluations, shadow a sample of production queries, compare latency and quality by slice, then shift traffic gradually.

The same discipline applies to generated summaries and hypothetical queries used during indexing. They are derived artifacts, not source truth. Keep lineage so they can be regenerated. For the conceptual reason model spaces behave this way, read why embeddings matter.

Dense, lexical, and hybrid retrieval

Dense retrieval maps queries and passages into vectors and ranks them by a distance or similarity function. It is strong when users paraphrase source language or ask concept-level questions. It can still rank a topically similar passage above the one containing the exact fact.

BM25 and related sparse methods reward lexical overlap while accounting for term frequency and document length. They remain especially valuable for:

• product SKUs and part numbers;
• stack traces and error codes;
• function, table, and configuration names;
• uncommon people or project names;
• quoted phrases and regulated terminology.

Dense retrieval is not “modern search” while BM25 is “legacy search.” They encode different signals.

Fuse rankings with Reciprocal Rank Fusion

Reciprocal Rank Fusion combines ranked lists without requiring their raw scores to have comparable scales. For document (d), a common form is:

[ \mathrm{RRF}(d) = \sum_{r \in R} \frac{w_r}{k + \operatorname{rank}_r(d)} ]

Here, (R) is the set of retrievers, (w_r) is an optional retriever weight, and (k) controls how sharply rank position matters. There is no universally correct (k), weight, candidate depth, or dense-to-lexical ratio. Tune them on your query distribution. Also define how missing documents, duplicate chunks, and ties are handled.

Hybrid retrieval is a strong candidate architecture because it preserves exact and semantic signals. It is not guaranteed to beat a tuned sparse retriever in every domain. SemEval-2026 Task 8 found widespread use of hybrid retrieval and query rewriting, while two of its top three retrieval teams used sparse methods; the organizers also cautioned that benchmark construction may have favored sparse retrieval. That is evidence for evaluation, not for copying one recipe.

Apply metadata and ACL filters before the model

Metadata turns a nearest-neighbor index into an application retrieval system. Useful fields include tenant, product, region, language, source type, lifecycle state, effective date, confidentiality class, and ACL principals or groups.

Separate relevance from eligibility

First determine which records the caller is eligible to see; then rank eligible records by relevance. If the storage engine supports secure pre-filtering, use it. If it uses approximate search with filters, test recall under realistic tenant sizes and ACL selectivity. Some implementations retrieve globally and filter afterward, which can return too few candidates and may expose unauthorized text to application components.

Authorization must be enforced before passages enter a reranker, prompt, log, cache, or analytics event. The model is outside the trust boundary for this decision.

Design cache keys around authorization

A response cache keyed only by normalized question can leak one user's answer to another. Include tenant, permission version or security scope, corpus/index version, locale, and relevant generation settings. Invalidate or bypass entries when permissions or source documents change.

For highly dynamic ACLs, storing every principal on each chunk may be impractical. Retrieve through document-level security joins, group-based claims, or a permission service—but preserve fail-closed behavior and trace the authorization decision.

Query rewriting for multi-turn conversations

The user asks, “Does it support rotation?” Search receives no explicit subject. The answer depends on conversation history, but sending the full transcript directly to retrieval introduces irrelevant terms and old intents.

Produce a standalone retrieval query

A rewrite stage can resolve references and produce “Does the Acme API access token support automatic rotation?” Keep the original question as a separate signal. Rewriting can accidentally change intent, invent constraints, or erase exact terms, so retrieve with both original and rewritten forms when the cost is justified.

Log the rewrite, model version, and conversation turns used. Treat sensitive conversation history under the same data controls as source documents.

Classify before expanding

Not every query should be rewritten. Exact identifiers may degrade when a model “corrects” them. A lightweight classifier can choose among pass-through, spelling normalization, standalone rewrite, decomposition, or clarification. Ambiguous requests should trigger a question rather than speculative expansion.

SemEval-2026 Task 8 specifically examined multi-turn RAG with unanswerable, underspecified, non-standalone, and unclear questions. Its results make query rewriting a serious design candidate, not proof that every turn needs an LLM call. Evaluate rewrite fidelity and downstream retrieval together.

Reranking: better ordering at a price

First-stage retrieval should maximize the chance that relevant evidence enters a manageable candidate pool. A reranker then spends more compute scoring query-passage pairs with richer interaction.

Cross-encoders and model rerankers

A cross-encoder reads query and passage jointly and often distinguishes fine-grained relevance better than independent embeddings. It adds latency proportional to candidate count and passage length. Batch requests, cap lengths deliberately, and avoid sending redundant chunks.

LLM rerankers can reason over nuanced criteria and multiple passages, but cost, latency, nondeterminism, and prompt-injection exposure are higher. They are easier to justify for low-volume, high-value workflows than for every interactive query.

Preserve diversity and authority

Pure relevance ranking may return six near-identical chunks from one document. Add deduplication or diversity constraints, and incorporate authority and freshness through explicit rules or features. Do not hide business policy inside an unexplained score multiplier.

Measure whether reranking improves nDCG, MRR, or downstream answer quality enough to justify its p95 latency and cost. Candidate count and cutoff should be domain-specific. A reranker cannot recover a relevant passage excluded by first-stage retrieval.

Assemble context for citations and abstention

Retrieval output is not yet a good prompt. Context assembly decides what the generator actually sees.

Build an evidence package

For each passage, include a stable citation ID, source title, canonical URL or document reference, source revision, effective date, and bounded text. Order passages intentionally, group related chunks, remove near-duplicates, and preserve headings. Never concatenate untrusted text into system instructions.

Use a token budget with reserved capacity for the question, instructions, and answer. A greedy “highest score until full” strategy can omit complementary evidence. Consider coverage across subquestions, authority, source diversity, and parent sections.

Make citations verifiable

Require claim-level citation markers that map to supplied passage IDs. Validate after generation that cited IDs exist and, for critical applications, check that cited text supports the claim. A citation is provenance, not proof of entailment. Linking a correct document to an unsupported sentence is still a failure.

Design abstention as a valid result

The system should say when evidence is absent, contradictory, stale, or outside the user's access. An answerability component can combine retrieval coverage, reranker scores, source authority, contradiction signals, and model assessment. Any threshold must be calibrated by risk and query type; a universal similarity cutoff is not defensible.

Provide a useful next action: request clarification, name missing information, link available sources, or route to a human. Evaluate abstention precision and recall, not only answer accuracy.

pgvector or a dedicated vector database?

The choice is not a contest between “simple” and “scalable.” It is a decision about workload, team capability, consistency, and search features. The vector database guide covers the underlying concepts; the matrix below frames an operational decision.

Benchmark with realistic vectors, filters, update rates, tenant distribution, and concurrency. Small uniform tests hide the behavior of selective filters, index churn, and skewed tenants. Include backup restore time, rolling upgrades, observability, and operator experience in the decision—not only median query latency.

Build an evaluation set before tuning

An evaluation set is a product artifact. Start with real questions from support, search logs, tickets, interviews, and domain experts. Remove or protect sensitive data. Add synthetic cases to improve coverage, but do not let generated questions replace real user language.

Label evidence, not only answers

For each query, capture relevant passages or documents, expected answer facts, acceptable alternatives, unanswerability, source authority, and criticality. Some queries have multiple valid evidence sets. Binary labels may be insufficient; graded relevance supports nDCG and better reflects partial usefulness.

Create slices for exact identifiers, paraphrases, multi-hop questions, tables, recent updates, multilingual queries, ambiguous requests, access-controlled content, and known failure patterns. Split development and held-out test sets so repeated tuning does not overfit the scorecard.

Measure retrieval directly

Recall@K asks what fraction of known relevant items appeared in the first K results. It matters when missing any supporting evidence is costly.

Precision@K asks what fraction of the first K results are relevant. It captures context pollution, although binary relevance can oversimplify partially useful chunks.

MRR uses the reciprocal rank of the first relevant item and is useful when one good result should appear early.

nDCG@K rewards highly relevant results near the top and supports graded relevance. It is valuable when several results have different usefulness.

K is part of the metric definition, not a universal operating setting. Report document-level and chunk-level metrics when appropriate: ten relevant chunks from one document can look good at chunk level while providing poor source coverage.

Evaluate generation and grounding

Measure whether the answer addresses the question, whether its claims are supported by supplied evidence, whether citations are correct, and whether it abstains appropriately. “Faithfulness” and “answer relevance” are useful categories, but implementation details differ across frameworks.

The Redis RAG evaluation guide, updated in June 2026, describes context precision, context recall, faithfulness, and answer relevance, and recommends offline regression evaluation. It also notes that synthetic generation is a baseline, not a replacement for user data and human labels. That distinction matters: a system can perform well on clean generated questions and fail on terse production queries.

Use LLM judges with controls

LLM judges make broad evaluation affordable, but they are not ground truth. They can prefer verbose answers, share biases with the generator, miss domain-specific errors, and change after provider updates. Pin judge versions and prompts, randomize answer order for pairwise comparisons, include human-labeled calibration sets, and measure agreement by slice.

Use deterministic checks where possible: citation IDs, JSON schemas, exact identifiers, forbidden claims, and ACL leakage do not require a judge. For high-risk domains, domain experts must review critical samples. Every release gate and target threshold should reflect business risk, label quality, and observed variance.

Engineer latency, cost, and caching budgets

End-to-end latency is the sum of authentication, rewriting, embedding, retrieval, fusion, reranking, context assembly, model queueing, generation, and network time. A p50 target alone conceals queue saturation and long-tail dependencies.

Allocate a stage budget

Define p50, p95, and timeout behavior for each stage. Trace whether a slow request came from the vector engine, reranker batch, LLM provider, or oversized context. Run load tests with production-like filters and prompt sizes. Graceful degradation might skip expansion, use a smaller candidate pool, bypass reranking, or return source links—but only when evaluation shows the fallback is safe.

Track cost per successful answer, not merely cost per token. Include embedding updates, duplicate indexing, reranking, evaluation jobs, cache infrastructure, and engineer operations. A cheaper generator that produces more escalations may increase total cost.

Cache at controlled boundaries

Candidate caches can reuse retrieval for stable normalized queries. Embedding caches avoid repeated query embedding. Semantic response caches can save the most but carry the highest correctness risk because similar wording does not guarantee identical intent.

Every cache needs a versioned key, TTL rationale, authorization scope, freshness behavior, and observability. Cache negative or abstained answers carefully: they can outlive an indexing repair. Measure hit rate alongside stale-hit rate, incorrect-hit rate, latency saved, and cost saved.

Make every answer traceable

Production RAG needs traces that cross service boundaries. One query trace should connect:

• authenticated tenant and security scope identifiers;
• original, normalized, and rewritten queries;
• parser, chunking, embedding, index, reranker, prompt, and model versions;
• filters and authorization outcomes;
• candidate IDs, ranks, and stage scores;
• selected context and token counts;
• citations, abstention reason, latency, and cost;
• cache decisions and user feedback.

Do not log raw sensitive text by default. Prefer stable IDs, hashes, redacted samples, access-controlled debug storage, and explicit retention. Sampling should retain errors, abstentions, long-tail latency, and security events at higher rates than routine successes.

Dashboards should connect service health with quality proxies: zero-result rate, source freshness lag, candidate overlap, reranker score distribution, citation coverage, abstention rate, cache hit rate, and feedback. Distribution shifts can warn that a model, corpus, or user population changed even before labels arrive. See AI observability for broader monitoring patterns.

Reindexing and migrations without downtime

Reindexing is a normal operation, not an emergency script. Any change to parser, chunker, embedding model, metadata schema, or distance function can require a new artifact.

Use immutable versions and aliases

Write a new index version beside the active one. Record expected document and chunk counts, failed items, checksum summaries, source watermark, and embedding manifest. Run validation before making it queryable. An alias or routing layer can shift reads without changing clients.

Compare before switching

Evaluate the new index offline, replay representative queries, and shadow production traffic without exposing answers. Compare result overlap, retrieval metrics, latency, filter behavior, and resource use. Canary by tenant or traffic percentage, monitor, then promote.

Keep the previous index until rollback risk is acceptable. During long builds, capture source changes through a queue or watermark and replay the delta before cutover. Test deletion propagation explicitly; a migration that restores added documents but resurrects deleted ones is not correct.

Lessons from larger pipelines are discussed in production RAG at scale.

Secure the retrieval and generation path

RAG expands the attack surface because untrusted source text can influence a model with tools and access to internal data.

Defend against indirect prompt injection

A retrieved page can contain “ignore prior instructions” or disguised tool commands. Treat passages as quoted data, isolate them from system and developer instructions, and tell the model that source content cannot grant permissions. Instruction hierarchy helps but is not a security boundary.

Tool authorization must be enforced outside the model with least privilege, typed arguments, allowlists, and confirmation for consequential actions. A cited document should never be able to cause a write merely by being retrieved. The same principle applies when RAG is exposed through MCP in production.

Control PII and sensitive data

Classify sources before indexing. Minimize stored text and metadata, encrypt in transit and at rest, restrict debug access, and set retention policies for queries and traces. Redact or tokenize sensitive fields when retrieval does not require the raw value.

Test cross-tenant leakage, ACL changes, cache isolation, deleted-user access, malicious documents, citation exfiltration, and model-provider boundaries. Maintain a kill switch for compromised sources or features. Security evaluation belongs in release gates, not a prompt appendix.

Recognize common incident patterns

Operational failures tend to repeat. A small incident taxonomy speeds diagnosis.

Quality drops after an embedding upgrade

Symptoms include rank-distribution changes, lower overlap, and failures concentrated in one language or content type. Verify that query and document vectors use the same model and normalization, then compare by evaluation slice. Roll back through the index alias if necessary.

Correct documents exist but are never retrieved

Check source watermarks, parser rejects, chunk lineage, lexical indexing, filter selectivity, and ACL joins before changing the prompt. A source may be indexed but excluded by an incorrect lifecycle date or tenant field.

Answers are stale after a policy update

Trace source modification to index availability, inspect caches, and confirm old revisions were superseded or deleted. Increasing top-K may make the contradiction worse by supplying both versions.

Latency spikes only for some tenants

Look for highly selective filters, large ACL expansions, tenant skew, cold partitions, oversized context, and reranker candidate inflation. Aggregate latency can hide one tenant's pathological query plan.

Citations point to the right document but wrong claim

Inspect context assembly and generation rather than retrieval alone. Enforce claim-level citations, preserve span IDs, and add support checks for high-risk claims.

For every incident, retain the trace and convert it into an evaluation case. The most valuable test set is partly a compressed history of production failures.

A staged 90-day implementation plan

The schedule below is a sequence, not a promise. Team size, regulation, corpus quality, and existing infrastructure determine pace.

Days 1–30: establish evidence and a baseline

Inventory sources, owners, authority, ACLs, and freshness objectives. Choose one bounded use case with measurable user value. Build a representative evaluation set and label retrieval evidence. Implement deterministic ingestion, stable IDs, parser failure reporting, one structure-aware chunking policy, and baseline lexical and dense retrieval.

Ship an internal interface that shows retrieved passages and provenance before polishing answers. Add end-to-end traces and a basic threat model. Record baseline Recall@K, nDCG@K, latency, cost, freshness lag, and abstention behavior. Avoid tuning a generator while retrieval remains invisible.

Days 31–60: improve retrieval and controls

Add hybrid fusion, metadata filtering, pre-model ACL enforcement, query rewriting for the slices that need it, and a reranker experiment. Compare each component through ablations: baseline, plus hybrid, plus rewrite, plus rerank. Keep changes only when the quality gain justifies complexity and tail latency.

Implement bounded context assembly, claim-level citations, answerability behavior, cache isolation, and offline generation evaluation. Run prompt-injection and cross-tenant tests. Establish version manifests and parallel index builds.

Days 61–90: prove operations

Run load tests, failure injection, source-lag drills, reindex and rollback exercises, and a limited canary. Build dashboards and alerts around freshness, retrieval failures, latency, citation coverage, security outcomes, and cost. Define incident ownership and a feedback-to-evaluation workflow.

Release to a controlled user group with explicit limitations. Review failed and low-confidence sessions weekly. Expand only after the team can explain regressions, restore a prior index, revoke content, and measure whether users complete the intended task.

Frequently asked questions

Production RAG is an engineering discipline

A reliable RAG system does not begin with a clever prompt and end with a vector query. It begins with source authority and freshness, preserves structure through ingestion, combines retrieval signals deliberately, enforces permissions before model access, assembles auditable evidence, and measures every stage.

The architecture should make change safe. Models, chunking strategies, indexes, and rerankers will improve. Versioned artifacts, representative evals, traces, parallel migrations, and rollback controls let a team adopt those improvements without guessing in production.

The practical standard is simple: for any answer, engineers should be able to determine which source revision was eligible, how it was retrieved, why it was selected, what the model saw, which claims were cited, what the request cost, and whether the behavior improved over the previous release. If the system cannot answer those questions, it is still a demo.

References

• Redis. How to evaluate RAG systems: metrics, frameworks & infrastructure. Published January 13, 2026; updated June 1, 2026.
• Sifei Meng and Dmitry Ilvovsky. Sifei at SemEval-2026 Task 8: Hybrid Retrieval and Query Rewriting for Multi-Turn RAG. Proceedings of the 20th International Workshop on Semantic Evaluation, 2026.

Related cluster notes

• Golden datasets for RAG eval
• Hybrid search and RRF
• Reranking in production
• Chunking experiments

Need this on your systems?

If you want a production vertical slice—RAG, agents, MCP tools, or an LLM gateway with budgets—see the AI implementation service.

Related: AI guardrails, prompt injection defenses, evaluation harness.