Mastering RAG: How To Architect An Enterprise RAG System

Source: Mastering RAG: How To Architect An Enterprise RAG System, Pratik Bhavsar, Galileo Labs, January 23, 2024.

A demonstration RAG system can be built in an afternoon: split a few documents into chunks, embed them, retrieve the nearest passages, place them in a prompt, and ask a language model to answer. An enterprise RAG system is a different kind of problem. It must identify the user, enforce tenant and document boundaries, reject unsafe input, rewrite incomplete questions, select an appropriate embedding model, ingest heterogeneous files, keep indexes current, choose a vector database under real cost and latency constraints, improve retrieval, validate the generated output, learn from feedback, expose traces, cache expensive work, and remain observable after launch.

The central lesson is that retrieval quality is not created by the vector database alone. It is the product of a chain of decisions. A good answer can be lost when the source document was never ingested, when the right chunk was ranked below the top K cutoff, when the assembly step discarded it, when noise prevented the model from extracting it, or when the prompt demanded the wrong level of detail. Building for production means designing the whole chain and measuring where it fails.

This guide starts with evidence from three RAG deployments, turns that evidence into seven failure modes, and then walks through the components of an enterprise architecture. The final sections focus on retrieval improvements, output control, continuous evaluation, and multi-tenancy, which are often treated as afterthoughts even though they determine whether a RAG product is safe to operate.

What the case studies reveal

A 2024 study examined three RAG systems in different domains: a cognitive reviewer for scientific research, an AI tutor for education, and a biomedical question-answering system. The systems did not share a domain, a document format, or a user population, but their failures rhymed. That is useful because it suggests the problems are architectural rather than accidental bugs in one application.

System and domain	Source material	Scale in the study	RAG stages involved	Representative questions
Cognitive Reviewer Scientific research	Research PDFs of varying size	Any collection size	Chunker, ranker, rewriter	What are the key points covered in the uploaded papers?
AI Tutor Education	Videos, HTML, and PDFs	38 documents	Chunker, rewriter, retriever, reader	What topics were covered during week 6?
BioASQ Biomedical question answering	Biomedical scientific PDFs	4,017 documents	Chunker, retriever, reader	Does pemetrexed cause hair loss? How is it treated?

Cognitive reviewer

The cognitive reviewer helps researchers analyze a collection of papers. A researcher defines an objective or research question and uploads related documents. The system ranks the papers against that objective so the researcher can review the most relevant material first. It also supports direct questions over the complete document set. The interesting design choice is that the first output is not a synthesized answer but a ranked reading surface. The system uses retrieval to direct human attention before it is trusted to generate conclusions.

AI tutor

The AI tutor answers student questions about a course unit using the unit's learning materials and exposes citations so students can inspect the evidence. It was integrated into Deakin University's learning management system and indexed PDFs, videos, and text documents. Video content was transcribed with Whisper before chunking. A query rewriter generalized follow-up questions, and the chat interface used prior dialogue to interpret each new request. This is a useful example of modality normalization: whichever interface produced a source, the downstream system still needs text or a searchable representation of its content.

Biomedical question answering

The biomedical system used the BioASQ dataset, which contains questions, links to supporting documents, and expert-prepared answers. The answer types included yes/no, summary, factoid, and list responses. That variety matters. A pipeline optimized only for fluent prose may fail when the user needs a binary determination, a ranked list, or a factual span. The expected output format is part of the retrieval and generation specification, not a formatting preference added at the end.

Seven failure points of RAG systems

The seven failure points form a useful diagnostic sequence. Each point asks where the required evidence was lost: did it exist, was it retrieved, was it placed in the model's context, was it extracted, and was it expressed in the form and specificity the user needed? This is more actionable than treating every bad answer as a generic hallucination.

Failure point	What went wrong	Typical cause	Production response
FP1: Missing content	The corpus does not contain the answer.	Incomplete ingestion, inaccessible sources, or a question outside the system's scope.	Detect unanswerable questions and say so; close ingestion gaps instead of encouraging the model to guess.
FP2: Missed the top-ranked documents	The answer exists, but its chunk falls below the top K cutoff.	A weak encoder, ambiguous query, poor chunk boundaries, or inappropriate K.	Measure recall at K, test query rewriting and hybrid retrieval, rerank candidates, and tune K against evidence.
FP3: Not in context	The document was retrieved but lost during consolidation or context assembly.	Too many documents, an aggressive compression strategy, or a selection rule that removes the useful passage.	Trace the exact context sent to the model; budget context deliberately and preserve source-to-prompt lineage.
FP4: Not extracted	The answer is in the context, but the model misses it.	Noise, conflicting passages, weak instructions, or a generator that cannot distinguish evidence from distractor text.	Clean and deduplicate context, add grounded extraction prompts, and evaluate the generator separately from retrieval.
FP5: Wrong format	The facts are present, but the response ignores the requested table, list, or schema.	Format requirements are absent from the prompt, are ambiguous, or are not tested.	Specify a schema, validate it programmatically, and add format-specific evaluation cases.
FP6: Incorrect specificity	The answer is either too general or too narrow for the user's context.	The system cannot infer the intended audience or the user asks a vague question with a hidden objective.	Clarify the task, carry reliable conversation context, and separate evidence from audience-specific explanation.
FP7: Incomplete	The response is accurate but leaves out requested information that was available.	A compound question asks for several items at once, or the model stops after the first supported answer.	Decompose multi-part questions, track required sub-answers, and evaluate completeness explicitly.

These failures point to a practical evaluation rule: do not score only the final answer. Record whether the required evidence was indexed, whether it was present in the candidate set, whether it survived ranking and assembly, and whether the generator extracted it. Without that intermediate visibility, a team may change the prompt when the real defect is a parser, or replace the vector database when the real defect is an authorization filter.

The enterprise RAG architecture

An enterprise architecture connects two flows. The indexing flow acquires documents, parses and enriches them, splits them into chunks, computes embeddings, and writes searchable records. The query flow authenticates the user, checks the input, rewrites the question, encodes it, filters and retrieves candidates, reranks and assembles context, generates an answer, validates the output, and records the interaction for feedback.

flowchart LR
    U[User] --> A[Authentication and tenant scope]
    A --> G1[Input guardrails]
    G1 --> Q[Query rewriter]
    Q --> E[Encoder]
    E --> R[Tenant-filtered retrieval]
    R --> RR[Rerank and diversify]
    RR --> C[Context assembly]
    C --> L[LLM generator]
    L --> G2[Output guardrails]
    G2 --> O[Answer with citations]
    O --> F[Feedback and traces]

    D[Documents and media] --> P[Parser and OCR]
    P --> CH[Chunker]
    CH --> V[Embedding and indexing]
    V --> R
    P --> S[Raw document storage]
    F --> M[Evaluation and improvement]
    M --> P

The diagram is a useful control map, but it should not be read as a promise that one vendor supplies every box. Authentication, storage, indexing, serving, evaluation, and observability may be separate systems with separate failure modes. The architecture is sound only when identity, access scope, and document lineage survive every boundary.

Authentication and access control

Authentication is the first component because an anonymous retrieval system cannot safely decide which documents a request may see. In an enterprise deployment, authentication supports six related goals:

• Access control: only authorized users can use the system and perform permitted actions.
• Data security: confidential information is not exposed to unauthorized people or processes.
• User privacy: personal information and account details remain visible only to the intended user.
• Legal compliance: regulated organizations can demonstrate that access controls match their obligations.
• Accountability: actions can be traced to a user for auditing and incident investigation.
• Personalization: the system can adapt content and preferences without mixing users together.

Managed identity services such as AWS Cognito or Firebase Authentication can provide sign-up and identity verification, but authentication alone is not authorization. The retrieval layer still needs a trustworthy tenant and permission scope derived from that identity. A user ID in a chat prompt is not an access-control mechanism because the model can repeat it, ignore it, or be manipulated by retrieved text.

Input guardrails

Input guardrails protect the application before a request reaches retrieval or generation. Large language models can be jailbroken, and an enterprise endpoint also faces data leakage, injection, resource exhaustion, and policy-abuse risks. The source guide recommends controls for several distinct cases:

• Anonymization or redaction: remove personally identifiable information such as names, addresses, and contact details before storage or external processing.
• Restricted substrings: block patterns that can enable SQL injection, cross-site scripting, or other injection attacks.
• Restricted topics: refuse hateful, discriminatory, explicit, or otherwise disallowed subject matter.
• Restricted code: prevent executable code from entering paths where it could compromise the service.
• Language checks: verify that the input uses a supported language or script when the downstream models require it.
• Prompt-injection detection: identify attempts to override instructions or make retrieved content control the system.
• Token limits: cap input size to avoid resource exhaustion and denial-of-service behavior.
• Toxicity detection: block abusive or harmful language and record the policy decision.

Llama Guard is one option for policy classification, either self-hosted or through a managed service, but it should not be treated as a perfect toxicity detector. A guardrail needs calibration, a defined failure policy, and monitoring for both false positives and false negatives. The application must also treat retrieved documents as untrusted input. A malicious document can carry an instruction injection even when the user's question is benign.

Query rewriting

Users often write queries that are short, ambiguous, or dependent on earlier turns. A query rewriter translates that request into retrieval-friendly text. The original guide describes three common techniques.

Rewrite from conversation history

A conversational query may omit the subject because both people know what they are discussing. Given the sequence "How many credit cards do you have?", "Are there any yearly fees for platinum and gold cards?", and "Compare features of both", the retrieval system needs to resolve "both" before it can search. The rewritten query becomes "Compare features of platinum and gold credit cards."

This step must distinguish durable preferences from temporary context. If the rewriter injects the wrong prior subject, it can make retrieval confidently irrelevant. Production systems should preserve the original query, the rewritten query, and the history window used for the rewrite so failures can be replayed.

Create subqueries

Compound questions are difficult to satisfy with one vector search. "Compare the features of platinum and gold credit cards" contains two entities and a comparison operation. A subquestion query engine can turn it into separate lookups such as "What are the features of platinum credit cards?" and "What are the features of gold credit cards?" The final generation step then compares the two evidence sets.

Subqueries also address FP7. If a user asks for the key points in documents A, B, and C, one retrieval call may overrepresent one document. Separate retrieval and completeness checks make omissions visible.

Create similar queries

Semantic and lexical retrieval can both miss a relevant document because the user chose different wording. The rewriter can generate paraphrases or domain-specific alternatives. "I want to know about platinum credit cards" may become "Tell me about the benefits of platinum credit cards." The candidate results can be fused or evaluated separately, but the system must deduplicate them and retain the source that actually supports the answer.

Choosing an encoder

The encoder converts documents and queries into vectors. It is one of the most consequential choices in the system because the same model must represent both sides of retrieval. A strong benchmark score is only the beginning of the decision.

Use MTEB as a starting point

The Massive Text Embedding Benchmark provides a broad comparison of embedding models across tasks, dimensions, retrieval performance, and model size. It can narrow the search and reveal open models that are competitive with proprietary APIs. It cannot prove that a model is best for a particular corpus. Benchmark training data may not match the enterprise domain, model documentation may be incomplete, and aggregate scores can hide weak performance on the exact document type that matters.

Treat MTEB as a screening instrument. The final shortlist should be tested on representative documents and real user questions.

Build a custom evaluation

The source describes three ways to evaluate candidate encoders on the actual domain:

1. Annotation: create a dedicated retrieval dataset, collect gold relevance labels from experts, and compare models with metrics such as Mean Reciprocal Rank and Normalized Discounted Cumulative Gain.
2. Model-assisted evaluation: generate candidate rankings with an LLM or cross-encoder, compare the candidate encoders, and manually inspect the strongest few when annotated data is too expensive.
3. Clustering: measure how much data is grouped at different Silhouette scores and experiment with algorithms such as HDBSCAN. The distribution can reveal whether an embedding separates the domain's concepts in a useful way.

These methods answer different questions. Annotation measures relevance, clustering exposes structure, and model-assisted ranking is often a faster way to find candidates for human review. None should be used without checking examples.

Evaluate operating tradeoffs

Encoder selection is also a systems decision:

• Query cost and availability: a hosted API removes model-serving work but creates a dependency; an open model requires CPU or GPU capacity appropriate to its size.
• Indexing cost: every document must be encoded, and re-encoding a large corpus can be expensive. Store embeddings separately so a vector database can be rebuilt without recomputing unchanged vectors.
• Storage cost: vector storage scales with dimension and corpus size. Estimate the average number of indexed units per document before committing to a large-dimensional model.
• Search latency: semantic search latency grows with embedding dimension, so a larger vector is not automatically a better retrieval investment.
• Language support: choose a multilingual encoder or add a translation stage when users search across languages.
• Privacy and compliance: finance, healthcare, and other sensitive domains may rule out sending text to an external embedding service.

The encoder decision should therefore be recorded as an architecture decision: the candidate models, the evaluation set, the measured recall and latency, the hosting model, the privacy constraints, and the conditions under which the choice will be revisited.

Document ingestion and parsing

Document ingestion turns source material into searchable records. Each document is parsed, split into chunks, encoded, and indexed alongside its original content and metadata. If this pipeline loses a table, misreads a scanned page, or drops a section, no amount of prompt engineering can recover the missing evidence.

Document formats

The parser must handle PDFs, word-processing files, spreadsheets, HTML, and other formats used by the organization. It also needs to preserve meaningful embedded content such as links, multimedia references, annotations, tables, and headings. A text extractor that returns only a flat string may discard the structure that makes a financial table or technical specification understandable.

Table recognition

Reports, research papers, and operational documents often place critical facts in tables. The parser should extract cell values, headers, row and column relationships, and table-level metadata. Table Transformer is one model that can help identify table structure. The retrieval representation should preserve enough structure that a generator can cite the correct row rather than paraphrase a jumbled sequence of cells.

Image and OCR recognition

Scanned pages, diagrams, screenshots, and image-only PDFs require OCR. The extracted text should be associated with page and region metadata so a citation can point back to the original source. OCR errors should be measured, especially in domains where a single character changes a drug, part number, or numerical result.

Metadata extraction

Metadata is information about the document rather than its main prose: author, creation date, document type, keywords, access group, tenant, and other attributes. An NLP or OCR pipeline can extract some fields, while the ingestion system should add authoritative fields from the source system. Metadata supports filtering, ranking, deduplication, freshness rules, and access control.

Metadata quality is not a cosmetic concern. A tenant identifier that is missing or mutable can create a data leak. An outdated effective date can make a correct passage irrelevant. A document type that is inferred incorrectly can send a query to the wrong index through query routing.

Chunking

Chunking decides what unit can be retrieved. If chunks are too small, a passage may lack the context needed to answer the question. If chunks are too long, retrieval becomes less precise and the generator receives more noise. Summarization can reduce size and encoding cost, but it can also remove the exact detail the user needs.

The best boundary depends on the domain. Technical manuals may follow sections and subsections, source code may follow classes and function definitions, legal documents may follow clauses, and conversational transcripts may follow speaker turns. A fixed character count is easy to operate but often cuts through the semantic unit that gives a passage meaning.

A practical pipeline can store multiple representations: a small child chunk for precise retrieval, a larger parent section for generation, and metadata that links the two. This is the foundation of recursive retrieval and sentence-window techniques described later.

Indexer

The indexer maps chunks and embeddings to searchable locations and performs create, update, and delete operations. It is a data system with production obligations, not just a write endpoint.

• Scalability: indexing and retrieval must remain acceptably fast as the corpus grows.
• Real-time updates: frequently changing source systems require live ingestion without destabilizing query traffic.
• Consistency and atomicity: concurrent updates must not leave partially indexed or contradictory documents.
• Storage efficiency: large corpora require attention to index size, replication, and retention.
• Security and access control: only authorized services may create, read, update, or delete records.
• Monitoring and maintenance: failed indexing jobs, stale indexes, and resource bottlenecks need detection and repair.

Atomicity matters for user trust. If a policy document is updated and deleted in one operation but its old chunks remain searchable, the system may answer from an obsolete rule. An ingestion job should have a durable status, idempotent writes, a deletion protocol, and a way to verify that the searchable set matches the source of truth.

Separate storage for separate responsibilities

An enterprise RAG system creates several kinds of data with different access, retention, and performance requirements. Combining everything into one database can make the first deployment simpler while making later recovery and evaluation harder.

Data type	Typical store	Why it is needed	Operational consideration
Embeddings	SQL or NoSQL database	Reindex without recalculating every vector; preserve a recoverable representation.	Keep the encoder version and vector dimension aligned with the vector database.
Raw documents	Object store or NoSQL database	Reprocess, cite, inspect, and recover the original source.	Preserve source hashes, access labels, and deletion state.
Chat history	NoSQL database	Resolve references, personalize, audit, and evaluate conversations.	Apply retention and privacy rules to sensitive dialogue.
User feedback	SQL or NoSQL database	Find weak queries, compare retrieval paths, and feed regression tests.	Connect the feedback to the exact retrieved context and model version.

The separate embeddings store is a particularly useful safety net. A vector database can be replaced, reindexed, or rebuilt from stored vectors and documents. Without that separation, a migration may require paying the encoder API to process the same corpus again and may produce a different index if the model version has changed.

Vector database selection

The vector database is the retrieval engine, but "which vector database is fastest?" is not a complete evaluation question. The best choice depends on the corpus, query mix, access patterns, deployment constraints, and the team that must operate it.

Recall versus latency

Exact flat search provides a strong recall baseline but becomes expensive at scale. Approximate indexes such as HNSW, product quantization, ANNOY, and DiskANN trade recall and memory for speed or storage. The tradeoff must be benchmarked on the organization's own vectors and queries because aggregate vendor benchmarks rarely reproduce the same filtering and document distribution.

Cost and hosting model

Managed cloud databases usually bill for storage and query volume and remove infrastructure work. Self-hosting can offer more control and potentially lower marginal cost, but the organization must own scaling, upgrades, backup, security, and incident response. Data sensitivity and team capability can matter more than a small difference in query latency.

Insertion speed versus query speed

Streaming systems need fast insertion, while user-facing search usually prioritizes stable query latency. Measure both under realistic peak load. A database that indexes quickly but expands latency when filters are present is not suitable for a permission-heavy enterprise interface.

Memory versus disk

In-memory indexes are fast but can become expensive or impossible when the vector set exceeds available RAM. Disk-based or memory-mapped systems can scale beyond memory, while DiskANN and related indexes target efficient out-of-memory search. The operational cost of memory must be included in the comparison.

Hybrid search

Vector search captures semantic similarity but may miss exact names, identifiers, and rare terms. Full-text search captures lexical matches but may miss paraphrases. Hybrid search combines dense and sparse retrieval, usually by maintaining a vector index, an inverted index, and a reranking or fusion step. Systems such as Pinecone, Weaviate, and Elasticsearch expose weights for balancing dense and sparse signals.

Metadata filtering

Enterprise queries commonly require filters such as tenant, department, document type, date, or permission group. Pre-filtering can exclude relevant candidates if the filter is applied too aggressively. Post-filtering can fail when the matching subset is tiny because the approximate search may return too few eligible records. Custom filtered search, such as Weaviate's approach, combines pre-filtering with a search structure that can still explore the filtered subset efficiently.

For security, tenant and permission filters are not optional ranking hints. They must be enforced before any candidate can enter the result set. A post-filter that sees private vectors in application memory is still a design weakness because the query path has already crossed a data boundary.

Techniques for improving retrieval

Longer context windows do not remove retrieval problems. Research cited by the source shows that models can be distracted by irrelevant passages and can miss material in the middle of a large context. The goal is not to send every plausible document; it is to send relevant, diverse, well-structured evidence.

Hypothetical document embeddings

HyDE asks a model to generate a hypothetical answer or document from the query, embeds that generated text, and uses the embedding to retrieve real documents. The hypothetical document can contain invented details, so it is not treated as evidence. Its value is that an answer-shaped passage may resemble the target corpus more closely than a short question does. This is especially useful when queries are brief or use vocabulary that differs from the source material.

Query routing

When the organization has multiple indexes, a router sends each question to the most relevant one. Product questions can go to product documentation, code questions to a code index, and policy questions to an HR or compliance index. Routing improves precision and latency, but the router's decisions need logging. A confident route to the wrong index creates a failure that may look like missing content even though the answer exists elsewhere.

Reranking

A first-stage vector retriever favors recall. A reranker then scores a smaller candidate set more carefully. Cross-encoder models such as BGE-large and decoder-based rankers such as RankVicuna, RankGPT, and RankZephyr can improve ordering and reduce hallucinations. The cost is additional latency and compute, so the candidate budget and reranker size must be measured. Reranking is most valuable when first-stage retrieval is broad enough to include the answer but not precise enough to place it near the top.

Maximal Marginal Relevance

Relevant results can be redundant. MMR balances relevance to the query with dissimilarity from results already selected. The effect is a result set that covers more facets of the question instead of returning several nearly identical chunks. It is useful for comparison questions, research synthesis, and multi-document answers where diversity improves completeness.

Autocut

Autocut decides how many results to return by looking for a significant jump in a ranked list of scores. In a distance example with values 0.1899, 0.1901, 0.191, 0.21, 0.215, and 0.23, the first three values form a tight group. An autocut setting can return that group, include a broader group, or return all six. The advantage is that K becomes data-dependent rather than a fixed number chosen for every query.

Recursive retrieval

Recursive retrieval, also called small-to-big retrieval, embeds smaller chunks for precise matching and then returns a larger parent context for generation. The small unit improves the chance that the relevant sentence is retrieved; the parent restores the explanation, table, or surrounding conditions needed to interpret it. This directly addresses the tension between retrieval precision and answer completeness.

Sentence-window retrieval

Sentence-window retrieval finds a highly relevant sentence and expands a window around it before generation. It is a lightweight version of the same idea: retrieve at a precise granularity and supply local context. The window should respect document structure so that it does not cross from one section into an unrelated one.

Choosing the generator

The generator turns retrieved evidence into an answer. The main choice is between a self-hosted inference service and a private API, but the correct comparison includes latency, throughput, security, observability, and the ability to detect hallucination.

API capabilities

A production LLM server should do more than accept a prompt. The source highlights four groups of capabilities:

• Performance: tensor parallelism across GPUs, continuous batching, quantization, and optimized transformer implementations improve throughput and latency.
• Generation control: temperature, top-p, top-k, repetition penalties, stop sequences, and log probabilities let the application control output and support hallucination analysis.
• Security: safetensors weight loading reduces model tampering risk, while watermarking can improve traceability.
• User experience: token streaming through Server-Sent Events makes responses feel responsive and allows the interface to show progress.

These capabilities are not interchangeable across providers. A model with excellent benchmark quality may be unsuitable if it cannot stream, expose log probabilities, enforce a stable timeout, or run inside the required privacy boundary.

Self-hosted inference

Self-hosting on AWS, GCP, Azure, or private infrastructure gives control over data and deployment, but it transfers responsibility for serving. TGI, Ray, and FastAPI-based stacks make different tradeoffs in operational complexity. Benchmark time to first token, inter-token latency, success rate, throughput, and correctness under load. Newer multi-adapter systems such as LoRAX can share GPU resources across many fine-tuned models when the product needs specialization without one deployment per model.

Private API services

Providers such as OpenAI, Fireworks, Anyscale, Replicate, Mistral, Perplexity, and Together remove infrastructure work but introduce rate limits, pricing changes, network dependencies, and data-governance questions. Compare input and output token prices with GPU utilization and staffing costs. A managed service may be cheaper at low or variable volume, while self-hosting may be more economical at sustained high utilization. The decision should be revisited as usage changes.

Prompting for grounded answers

Retrieval supplies evidence, but the prompt determines how the model uses it. The source points to five prompting patterns that can improve RAG behavior:

• Thread of Thought: break a complex question into smaller reasoning steps before answering.
• Chain of Note: take structured notes as the model works through a long context, then synthesize them.
• Chain of Verification: draft an answer, generate verification questions, and revise based on the checks.
• Emotional prompt: add an emotional emphasis to the instruction, which can change model compliance but is less reliable and less interpretable.
• Expert prompting: add a detailed expert role and task-specific framing to improve domain behavior.

For enterprise RAG, the prompt should require evidence-grounded answers, make uncertainty acceptable, define the output format, and tell the model what to do when the retrieved context is insufficient. Prompt instructions should not be the only enforcement mechanism. Citations, schema validation, and access checks must exist outside the language model.

Output guardrails

Output guardrails inspect the generated answer before it reaches the user or another system. They can detect hallucinated claims, unsupported specificity, competitor mentions, toxic language, policy violations, leaked sensitive data, and brand risk. The source uses an example in which a climate question elicits a scientifically grounded answer in one case and an insulting, dismissive response in another. The second response may satisfy a narrow definition of fluency while causing real brand and safety damage.

Output checks should be layered: groundedness against retrieved evidence, format and schema validation, policy classification, sensitive-data detection, and deterministic business rules. Each failure needs a defined action such as blocking, regenerating, redacting, escalating, or returning a safe fallback. Logging the raw response, the transformed response, and the reason for intervention is necessary to debug the guardrail itself.

User feedback and the improvement loop

Feedback turns production use into an evaluation source. A thumbs-up or thumbs-down, star rating, corrected answer, or free-text comment is only useful when it can be joined to the original query, retrieved chunks, ranker scores, prompt, model version, and final answer.

1. Collect feedback: provide low-friction positive and negative signals and allow targeted comments.
2. Diagnose the failure: inspect whether the issue came from missing data, retrieval, consolidation, extraction, format, specificity, or completeness.
3. Improve the data: correct incomplete or poorly organized source content, fix parsing, update access labels, or add missing documents.
4. Evaluate and add a regression test: rerun the weak query after the change and preserve it as a test so future changes cannot silently break it.

The most important improvement is often in the underlying data rather than the prompt. If a policy changed and the old document remains indexed, the system needs a deletion and freshness process. If a table is being flattened, the parser needs work. If the answer was retrieved but omitted, the assembly or completeness evaluation needs attention.

Observability

RAG observability must cover both conventional service metrics and model-specific behavior. Latency and cost are necessary; they are not sufficient. A fast, inexpensive response can still be ungrounded, incomplete, or outside the permitted data scope.

• Prompt analysis: inspect real prompts and outputs for instruction failures, hallucinations, and drift.
• Traceability: capture spans from frameworks such as LangChain and LlamaIndex so a slow or incorrect answer can be followed through retrieval, reranking, assembly, and generation.
• Retrieval diagnostics: track recall, score distributions, filter behavior, index freshness, and the effect of retrieval parameters.
• Alerting: detect error-rate increases, latency regressions, cost spikes, hallucination signals, and unusual access patterns.

Service-level objectives should include time to first token, end-to-end latency, availability, retrieval failure rate, groundedness, and cost per successful interaction. Tracing should record document IDs and access-scope decisions, but it must not create a second uncontrolled copy of sensitive content.

Caching

Caching stores prompts and responses, or reusable intermediate results, so the system does not repeat expensive work. It has three practical benefits:

• Faster production inference: repeated or similar queries can approach zero generated-token latency.
• Faster development: engineers can iterate on downstream logic without paying for the same model call on every test run.
• Reusable data: stored prompt-response pairs can support analysis and future fine-tuning, subject to privacy and retention rules.

GPTCache is one implementation that supports exact and similar-match caching and reports hit rate, latency, and recall. Similarity caching needs a strict tenant boundary. Two users may ask the same words while having different permissions or document access. A cache key that ignores tenant, authorization scope, document version, and model version can leak a previous answer or return evidence the current user is not allowed to see.

Multi-tenancy: isolation is an access-control property

Multi-tenant RAG has to satisfy two goals at once: retrieve the best evidence for the current user and prevent any other tenant's data from entering the query path. The simplest reliable design attaches authoritative tenant, user, and permission metadata to every document and chunk, then applies a mandatory filter before retrieval and ranking.

flowchart LR
    Q[User query] --> I[Authenticated identity]
    I --> S[Tenant and permission scope]
    S --> F[Pre-filter searchable records]
    F --> V[Vector and lexical retrieval]
    V --> R[Rerank only authorized candidates]
    R --> G[Generate from authorized context]
    G --> A[Answer]
    A --> L[Audit log]

The source recommends metadata-based isolation because it is simple and effective: include the tenant's details when documents are added, tie each document to that tenant, and filter search results by the same attribute. A smart search can then rank only the documents that belong to the user's tenant. The benefit is that private information does not mix between users.

The tenant filter must be enforced by the data-access layer, not requested from the language model. A prompt that says "only use documents for tenant A" is not a security boundary; retrieved documents and user messages can attempt to override it.

The strongest implementation uses tenant-specific namespaces or collections where operational cost allows them, plus filtered queries for shared collections. Either way, the authorization decision must be made before vectors, snippets, or context are returned. The audit record should show which identity requested the answer, which scope was applied, which document versions were eligible, and which sources supported the response.

There are additional isolation questions beyond the first filter. Can one tenant poison a shared cache? Can a re-ranker see candidates from another tenant? Can an evaluator export raw traces across accounts? Can an administrative indexer write into the wrong namespace? Can a citation endpoint be guessed and used to retrieve a document outside the user's scope? A multi-tenant design must follow tenant identity through all of these paths, not just the initial vector query.

Conclusion

An enterprise RAG system is a set of coupled information and control systems. Authentication establishes identity. Guardrails constrain unsafe requests and outputs. Query rewriting makes user language searchable. Encoders determine what similarity means. Parsers and chunkers determine what evidence can exist. Indexers and storage systems keep that evidence available and correct. Vector databases and retrieval techniques determine which evidence reaches the model. Prompts and generators turn evidence into an answer. Feedback, observability, caching, and multi-tenancy keep the system improving without weakening trust boundaries.

The seven failure points provide a durable debugging frame. When an answer is wrong, locate the loss before changing the model: missing content, missed ranking, lost context, failed extraction, wrong format, incorrect specificity, or incompleteness. That discipline separates a persuasive demo from an enterprise service that can be evaluated, audited, and operated.

Source note: The original article domain was unavailable when this archival edition was prepared. The text above was adapted from the complete Internet Archive capture dated October 11, 2024, and the original technical figures were recovered from archived Galileo CDN assets. The canonical source remains the original article URL.