RAG Architecture Playbook for Production Knowledge Systems

Build a RAG system with authorized retrieval, source lineage, measured answer support, access-revocation and deletion controls, and rehearsed index recovery.

trigger="Users need answers from changing private or domain-specific sources, with evidence and access appropriate to the active user." owner="The knowledge-product owner accountable for answer usefulness and the source-to-answer operating contract." participants={["Domain reviewer", "Data and source owners", "Identity and security owners", "AI engineer", "Retrieval engineer", "Operations lead"]} prerequisites={[ "An authoritative source inventory with rights, owners, permissions, update behavior, and retention.", "Representative questions, expected evidence, and unanswerable or restricted cases.", "A citation and abstention policy, an approved provider boundary, and a non-generative fallback.", "A feasible revocation and deletion path through indexes, caches, conversations, and recovery copies." ]} outputs={[ "A versioned source-to-answer architecture, permission contract, and lineage record.", "A retrieval and answer evaluation report with critical slices and known gaps.", "An ingestion, refresh, revocation, deletion, release, and recovery runbook." ]} doneWhen={[ "Negative access tests exclude unauthorized evidence from retrieval results, prompts, citations, and caches.", "Domain review accepts answer support and abstention on the defined evaluation slices.", "Source changes and access revocations meet agreed propagation objectives or fail closed.", "Operators can stop a corpus and recover a compatible release without restoring revoked or deleted content." ]} />

Treat retrieval as a knowledge system

Retrieval-augmented generation selects source evidence before a model answers a question. It can help with changing knowledge and identifiable references. It does not guarantee that the selected source is correct, current, permitted, or faithfully represented in the answer.

The original RAG research studies combining retrieval with generation on defined tasks. It is not a guarantee of results on an enterprise corpus or a complete production security design.

Separate four responsibilities: source lifecycle, authorization, retrieval quality, and answer evidence. A vector index is one implementation choice. Begin with a bounded source collection and a real user task rather than indexing every available document.

1. Define the task and answer policy

The product owner identifies the users, decisions supported, allowed source classes, consequence of a wrong answer, and expected fallback. Distinguish search, summarization, advice, and action execution. This playbook covers evidence-based answers; an answer must not acquire permission to execute a tool merely because it cites a document.

The domain reviewer defines how to handle conflicting editions, missing evidence, ambiguous questions, and unsupported claims. Choose whether the system should ask a question, provide a source link, produce a limited answer, or abstain.

Record the current baseline, such as keyword search or a manual lookup workflow. Measure successful task completion and reviewer correction effort as well as retrieval scores. A more elaborate architecture must justify its added latency, operating cost, and failure modes.

Gate: the team can describe a correct answer and a correct refusal for each critical task class before tuning the pipeline.

2. Inventory sources and preserve lineage

The data owner confirms the right to ingest and process each source, including provider disclosure, licensing, location, retention, and derived artifacts. Do not infer approval from the ability to download a document.

Each source record needs a stable ID, authoritative location, owner, version or content hash, effective date where relevant, access-policy reference, ingestion status, and deletion state. Each derived chunk needs its source and parent IDs, parser and chunking versions, embedding configuration, and index generation.

Test parsing on representative formats:

| Source type | Representation to investigate | Check before indexing | | --- | --- | --- | | Prose and procedures | Sections or parent-child units | Definitions, exceptions, and prerequisites stay with actions | | Technical documentation | Versioned headers and procedures | Product version and deprecated guidance remain identifiable | | Code | Symbols with repository and revision lineage | Access, imports, and surrounding interface context | | Tables | Header-aware rows or table regions | Units, row relationships, merged cells, and values | | Layout-heavy documents | Reading-order and page-aware blocks | Columns, footnotes, OCR errors, and citations | | Conversations | Bounded turns with reply context | Participant permissions, retention, and deletion policy |

Quarantine extraction failures instead of treating an empty parse as a successfully indexed source. Reconcile the source catalog with active artifacts and track failures by owner.

3. Enforce authorization independently of the model

The security owner defines document access from authenticated identity, tenant membership, source policy, and the operation being requested. The server obtains trusted policy context. User text, a query rewrite, or a model-generated filter cannot widen it.

Enforce the access constraint on every retrieval route, including lexical, vector, hybrid, multi-query, parent expansion, direct-ID fetch, and fallback. Restricted passages must not reach the reranker, generation provider, logs, or another tenant's cache. Re-check current authorization before context assembly and delivery where policy can change during processing.

Microsoft's security-filter pattern illustrates applying principal filters to search. Its filter strings do not authenticate a user. The application must supply trusted principals and apply the constraint consistently; merely hiding the policy field from a response is not access control.

"type": "svg-architecture", "title": "Only authorized evidence reaches the answer path", "nodes": [ ], "links": [ ], "caption": "This is the permitted path. Missing or denied policy stops processing at its gate; there is no unfiltered fallback. Retrieved content remains untrusted data." }} />

Define revocation behavior separately from ordinary content freshness. If indexed permissions can lag, use a current authorization check, a deny overlay, or disable the affected scope until synchronization is verified. Choose the mechanism from the source's policy guarantees and the harm of exposure.

Scope answer and retrieval caches to the full authorization context and relevant policy version. Revalidate when that context changes. Previously disclosed answers cannot be recalled from a user's memory, but future conversation display, exports, cache hits, and citation access still need a policy.

4. Build a versioned baseline retrieval path

The retrieval engineer begins with a small set of testable options. Compare lexical retrieval for exact identifiers and terms, vector retrieval for semantic matches, and a hybrid candidate when both matter. Do not assume that combining them always improves quality.

Select an embedding model on the target corpus and question set. Record the exact available model identifier, dimension setting, distance metric, normalization assumptions, language coverage, input limits, and provider constraints. Do not assign one dimension or a “best” ranking to an entire model family.

Document embeddings and query embeddings must use a compatible representation. A changed model, dimension, or preprocessing rule may require a new index. Mixing incompatible generations can produce invalid or misleading results even when an API accepts the request.

For the retrieval store, test permission-filter behavior, upsert and delete propagation, query latency under restrictive filters, capacity, export, and restore. Use vector database selection for a broader comparison rather than turning this guide into a vendor ranking.

Gate: the baseline retrieves permitted evidence with reproducible lineage and acceptable operational behavior.

5. Treat chunking and query expansion as experiments

Choose chunk boundaries from document structure and the questions users ask. Smaller units can improve specificity but omit conditions; larger units preserve context but add noise and token cost. Parent-child retrieval must authorize the parent independently and preserve source lineage.

Semantic segmentation based on sentence embeddings is another hypothesis. A similarity drop is not proof of a meaningful topic boundary. Evaluate the parser, boundary rule, and model together on difficult documents.

Keep the original question when evaluating rewrites. A rewrite can change a date, entity, negation, or task. For multi-query retrieval, execute each lexical or vector query with the same server-enforced access scope, then merge and deduplicate candidates while retaining their source identities. Embedding belongs before vector search, not after retrieval.

HyDE generates a hypothetical passage and uses its representation to retrieve source evidence. It can also reinforce an invented premise. Compare it against the baseline on ambiguous, incorrect, and unanswerable questions. Do not treat the hypothetical passage as a cited source or claim a universal retrieval uplift.

Change one strategy at a time where practical. Record failures as well as average improvement, including exact identifiers, multilingual terms, tables, dated policies, and negation.

6. Rerank and assemble coherent evidence

A reranker can improve ordering when the required evidence is already in the candidate set. It cannot recover an excluded source. Measure candidate recall before interpreting a better final ranking score.

Sweep candidate and context counts on the same evaluation set. Compare answer support, latency, token use, provider disclosure, and cost per resolved task. Keep only configurations whose benefits justify their operating burden.

Deduplicate without merging incompatible editions or erasing disagreement. Preserve the qualifications, table headers, units, and procedural prerequisites needed to understand a passage. If compression is used, evaluate the compressed result against its source and keep auditable lineage.

The Lost in the Middle study found position sensitivity under its tested settings. That supports testing evidence placement; it does not establish one reliable ordering rule for every current model. Include decisive evidence early, in the middle, and late in your evaluation.

For reranker failure, choose a tested first-stage ordering or abstention. Either fallback must preserve authorization and the task's minimum evidence standard.

7. Verify answers and contain hostile content

The AI engineer treats source text, retrieved instructions, and model output as untrusted input. A document can contain a command to disclose secrets or contact an external service. Delimit evidence, constrain tool authority outside the model, and test adversarial documents and queries.

OWASP's prompt-injection guidance describes both direct and indirect attacks. RAG and system prompts do not eliminate this risk. Do not place secrets or privileged action controls in a prompt and assume wording alone protects them.

Require identifiable source versions and locations for material claims. A citation is only useful if the passage supports the claim and the user is allowed to read it. Check contradictions, altered numbers, missing conditions, and unsupported conclusions.

Automated checks can validate citation IDs and flag suspicious claims, but semantic support can remain uncertain. Use domain review for consequential answers and abstain when the evidence standard cannot be met. Do not describe an automated verifier as a factual guarantee.

8. Evaluate retrieval and answers separately

The evaluation owner creates held-out cases with question, user and tenant context, allowed sources, expected evidence, answer rubric, and correct abstention behavior. Keep a frozen regression set and rotate reviewed cases to reduce tuning to one benchmark.

| Gate | Evidence | Release-blocking example | | --- | --- | --- | | Candidate recall | Required evidence among retrieved candidates | A critical procedure's prerequisite is missing | | Final context usefulness | Relevant and coherent evidence after assembly | Old and new policy editions are blended | | Claim support | Material claims reviewed against cited passages | A cited table does not support the answer's value | | Task usefulness | Domain rubric and completion outcome | Fluent answer addresses the wrong decision | | Access and abstention | Restricted, missing, conflicting, and hostile cases | Another tenant's passage or confident unsupported answer | | Operations | End-to-end load, lag, dependency, and cost tests | Quality improves while freshness or completion fails |

Report sample size, uncertainty, reviewer disagreement, and results by critical slice. Validate model-based judges against human-reviewed cases and version their configuration. A high aggregate score must not hide an access violation or a critical task failure.

Keep production tracing minimal and permission-controlled. OpenTelemetry's sensitive-data guidance supports reviewing collection and redaction. Raw questions and passages may create a new sensitive repository; use necessary identifiers and categories where they suffice.

9. Propagate changes, revocations, and deletions

The ingestion owner records source events with stable identity and sequence information where available. Processing must tolerate duplicates, delayed delivery, and restart. A late update must not recreate a source that has since been deleted.

"type": "flow", "title": "Change source state without reopening old access", "steps": [ ], "caption": "Ordinary content refresh can build a candidate version. Revocation or deletion needs an immediate serving control when asynchronous cleanup cannot meet the access objective." }} />

Maintain lineage for all derived artifacts: raw copies, chunks, parent views, embeddings, lexical indexes, generated summaries, cached answers, conversations, and exports. Define what must be removed, restricted, or retained under an approved obligation. A tombstone blocks serving but does not prove physical erasure.

Track cleanup checkpoints, failed work, and retained exceptions. Run negative tests through every serving route, not only a direct vector lookup. Reconcile the source catalog against active and standby generations.

Backups and old indexes need a restore policy. Before reopening a restored generation, replay current revocations and deletions and verify them. A rollback must not reintroduce content that is no longer permitted.

10. Release and recover the complete system

The release owner versions parser, chunking, embeddings, retrieval configuration, reranker, prompt, application, and source/index generation. Record provider identifiers and reproducibility limits where immutable snapshots are unavailable.

Build a candidate generation without silently overwriting the active one. Validate permissions, lineage, retrieval, answers, capacity, and deletion state. A switch across vector, lexical, metadata, and cache layers must use a compatible generation or a documented consistency gate.

| Failure | First safe action | Recovery gate | | --- | --- | --- | | Unauthorized evidence appears | Disable affected corpus or route; start security response | Correct policy, scope assessment, and negative access tests | | Source freshness falls behind | Stop claims requiring current data or use approved source fallback | Reconciled checkpoint and backlog within objective | | Candidate lacks expected evidence | Inspect parsing, filters, and representation | Known-answer regression cases pass | | Reranker or model fails | Use tested bounded fallback or abstain | Same access and minimum evidence standard | | New index loses quality | Restore a compatible prior manifest | Apply current permissions and deletion state before serving | | Partial deletion or late update recreates data | Deny affected source and stop unsafe processing | Idempotent cleanup, event-order tests, and reconciliation |

For a bounded pilot, define cohort, stop conditions, review capacity, and observation period from risk and traffic. Include cost of retries, abstentions, human corrections, and stale-index repair. The AI production playbook covers the broader release operating contract.

11. Reuse the source-to-answer record

Task, permitted use, accountable owner, and fallback:
Source IDs, rights, owners, editions, and authoritative policy:
Parser, chunking, embedding, retrieval, and prompt identifiers:
Index generations, lineage, and compatible cache policy:
Trusted identity path and authorization enforcement points:
Revocation/deletion objectives, serving block, and cleanup checkpoints:
Evaluation cases, allowed sources, rubric, and critical-slice results:
Provider boundaries, latency, cost, and failure-test evidence:
Pilot scope, stop conditions, reviewers, and incident contacts:
Recovery manifest, current deny state, and restore test:
Limitations, approvals, and next review trigger:

"Source rights, authoritative identity, and current permissions are recorded.", "All retrieval, fallback, parent, cache, and citation paths enforce the same access policy.", "Parsers preserve units, qualifications, editions, and source locations.", "Models and index configurations are recorded without vendor rankings or assumed uplift.", "Critical evaluation slices include unsupported, restricted, conflicting, and hostile inputs.", "Revocations and deletions have serving controls, cleanup evidence, and restore tests.", "A compatible release can recover without resurrecting revoked or deleted material.", "Named domain, data, security, and operations owners hold the remaining approval decisions." ]} />

Run the first production-readiness review

Bring one source owner, one domain reviewer, the identity owner, the retrieval engineer, the application owner and the operator who would respond to an incident. Review a single question from authentication through retrieval, context assembly, answer, citation, trace and cache. Then repeat the exercise with a restricted user, a deleted source, a conflicting version and an unanswerable question.

The review is complete only when each participant can identify the evidence they own and the control that stops unsafe processing. Capture the exact index generation, policy version, retrieval configuration, model and evaluation run. Record every exception with an owner and expiry rather than converting it into an informal launch assumption.

Start with a cohort whose questions and source access are understood. Keep the previous search or manual workflow available. During the pilot, sample supported answers, refusals, access denials and corrections. Track whether users complete the task, whether cited evidence is sufficient, how often reviewers intervene, and how much the complete path costs.

Stop expansion when permission behavior is uncertain, ingestion failures cannot be reconciled, critical evidence is missed, or the team cannot disable an affected corpus independently. Resume only after the failed control is corrected and the relevant negative tests pass against the release candidate.

Limitations

A corpus can contain incorrect source material, access changes may be hard to observe, and model behavior is not fully deterministic. State these limits in the product's answer and release policy. No retrieval score or citation checker certifies factual correctness.

Provider capabilities and source connectors change. Verify the deployed versions, data-handling terms, and policy semantics for the actual system. This editorially reviewed playbook is not domain approval or authorization to release a knowledge system.