Bedrock Knowledge Bases: Retrieval Configuration Trade-offs
Choose chunking strategy, embedding model, and retrieval settings for Amazon Bedrock Knowledge Bases with known trade-offs documented. Covers fixed-size,...
The choice that matters most when setting up a Bedrock Knowledge Base is not which vector store to use. It is how you chunk and embed your documents before the first query ever arrives. A chunking decision made during ingest determines the evidence your generation model will see at answer time. Getting it wrong means retrieval returns passages that are too small to carry context, too large to rank precisely, or semantically misaligned with the query. This article helps you make that decision with known trade-offs documented.
The article covers customer-managed knowledge bases, where you configure and manage the vector store (Amazon OpenSearch Serverless, Aurora PostgreSQL, or a custom store), chunking strategy, and embedding model directly. Amazon Bedrock also offers a Managed Knowledge Base type, in which AWS manages the ingestion, indexing, storage, and retrieval infrastructure; that type provides different capabilities including multi-modal ingestion and agentic retrieval, with less direct control over vector store selection and ingestion pipeline configuration. Both types share the same retrieval quality principles covered in this article. Source: Knowledge Bases overview, checked 2026-11-05.
What the configuration actually controls
A Bedrock Knowledge Base produces answers through a three-stage contract: ingest, retrieve, generate. Configuration decisions at each stage compound.
Ingest splits source documents into chunks, embeds them using the model you specify, and writes both the vector and the source text to your vector store. The chunking strategy determines what text is stored and how passages relate to each other. The embedding model determines the vector space.
Retrieve accepts a query, embeds it using the same model (the query and document encoder must match), searches your vector store, and returns a ranked list of text passages. The numberOfResults parameter controls how many candidates the retrieval stage surfaces before passing them to the generation stage.
Generate receives the retrieved passages and your user query, then produces an answer. If the retrieved passages do not contain the answer, the generated response will be wrong regardless of how well the model writes.
This means chunking and embedding decisions made at ingest time directly determine what evidence is available during generation. They cannot be changed in place: the Knowledge Bases UpdateKnowledgeBase API does not permit changing the embedding model configuration after creation. A changed chunking or embedding strategy requires a new knowledge base resource and a full re-ingest, as documented in the Knowledge Bases overview.
Chunking strategies available in Bedrock Knowledge Bases
As of the source-checked date, Bedrock Knowledge Bases supports four chunking approaches for text documents.
| Strategy | How it works | Best suited for |
|---|---|---|
| Fixed-size | Splits on token count with configurable overlap percentage | Consistent document structure; policy documents; FAQs |
| Default | Approximately 300 tokens per chunk, respects sentence boundaries | General-purpose; unknown document mix |
| Hierarchical | Child chunks (small, precise) embedded; parent chunks (larger, context-rich) returned at retrieval | Long documents where answer precision and context width both matter |
| Semantic | Uses an embedding model at ingest to group semantically coherent sentences | Narrative or editorial text with topic shifts mid-document |
A fifth option is no chunking, where each document is treated as a single chunk. This is only appropriate for very short documents and disables page-number citation metadata.
Fixed-size chunking is the most predictable option. You specify a maximum token count per chunk (the documentation does not state a hard upper limit; check current API constraints) and an overlap percentage. The overlap causes consecutive chunks to share a tail of tokens, which helps when a query answer spans a chunk boundary. The trade-off is that fixed boundaries ignore sentence and topic structure. A policy clause that starts near one boundary and concludes past the next will be split, and neither half may carry enough context to answer a question about the whole clause.
Hierarchical chunking addresses this by maintaining two levels: child chunks that are small and precise for embedding comparison, and parent chunks that are wider for context. At retrieval time, Bedrock returns the parent chunk when a child chunk matches. Because returned results may be parent-level, the actual numberOfResults returned can be fewer than requested when multiple children share a parent. The AWS documentation explicitly notes this behavior. This strategy is not recommended when using S3 vector bucket as your vector store due to metadata size constraints.
Semantic chunking uses a foundation model at ingest time to identify sentence boundary points by measuring semantic distance between adjacent sentences. The configurable parameters are maximum tokens, buffer size (how many surrounding sentences are included when measuring similarity), and a breakpoint percentile threshold that determines how dissimilar adjacent text must be before the chunker draws a boundary. Semantic chunking incurs additional cost because it invokes a model during ingestion. It produces the most topically coherent chunks but is the least predictable in chunk size and count.
Worked example: HR policy corpus
An HR policy corpus typically contains documents with distinct logical sections: eligibility criteria, process steps, exception handling, and contact information. These sections often appear within the same document and have very different answering characteristics. Consider this setup decision for a 200-document corpus of HR policies.
What the reader asks: An employee submits a question: "Am I eligible for parental leave if I am on a fixed-term contract?"
Why chunking matters: A policy document might contain an eligibility section that lists contract types, followed several paragraphs later by a detailed exception clause covering fixed-term employees. If fixed-size chunking splits these at a 300-token boundary between the two sections, retrieval returns either the eligibility list (without the exception) or the exception (without the eligibility criteria). Neither passage alone answers the question correctly.
Comparison of outcomes:
| Setting | Chunk shape | Retrieval result for "fixed-term parental leave" | Answer quality risk |
|---|---|---|---|
| Fixed-size, 300 tokens, 10% overlap | Uniform blocks; overlap catches some boundaries | May retrieve eligibility criteria without the exception clause | Incomplete answer; states no eligibility |
| Fixed-size, 500 tokens, 20% overlap | Wider blocks; more likely to contain both criteria and exception | Usually captures both in one chunk for shorter policies | Acceptable for most policies; oversized for FAQs |
| Hierarchical (child 200, parent 800 tokens) | Retrieves wide parent context when child matches | Parent chunk likely contains both sections | Good; slightly higher token cost per retrieved result |
| Semantic | Topic-aware boundaries | Treats eligibility and exception as one coherent section if adjacent | Strong; depends on ingest model cost and predictability |
For this corpus, hierarchical chunking with child chunks around 150-200 tokens and parent chunks around 600-800 tokens provides a reasonable starting point. Semantic chunking is worth evaluating for well-structured narrative policies, with the understanding that ingest cost scales with corpus size and model pricing.
After choosing a strategy, build a retrieval evaluation set. Select 15-20 representative questions and identify the source passage that answers each. Run retrieval and check whether that passage appears in the top-k results. Count how often it does. This hit rate, measured before generation, tells you whether the retrieval configuration can provide evidence at all. A generation model cannot improve a hit rate of zero.
Embedding model selection
As of source-checked date, Bedrock Knowledge Bases supports several embedding models including Amazon Titan Text Embeddings V2 and Cohere Embed v3. The choice has two practical consequences: vector dimensions and inference cost.
Vector dimensions affect precision and cost. Higher dimensions generally preserve more semantic nuance, but require more storage and can increase query latency at large scale. Amazon Titan Text Embeddings V2 supports configurable dimensions (256, 512, or 1024); lower dimensions cost less per embedding but may lose retrieval precision for domains with many semantically similar documents. Cohere Embed v3 distinguishes between search_document and search_query input types, which the query encoder must respect at retrieval time.
The same model must encode both documents at ingest time and queries at retrieval time. As of this writing, the UpdateKnowledgeBase API does not permit changing the embedding model after creation; verify this against current AWS Bedrock documentation before migration planning. Plan the embedding model choice before first ingest. If the limitation is still in effect and you need to change it, you will need to rebuild the knowledge base as described in the Bedrock Knowledge Base Rebuild playbook.
Do not evaluate embedding quality by dimension count alone. Run retrieval comparison on your actual domain questions. A 512-dimension model that scores well on your document set is preferable to a 1024-dimension model that does not.
Retrieval configuration at query time
After ingest, you can tune retrieval without rebuilding the knowledge base. The main lever is numberOfResults, which controls how many chunks the retrieval stage returns. More results give the generation model more evidence candidates, but also increase the prompt size and token cost.
A secondary lever available at query time is metadata filtering. If you have attached metadata to your source documents (for example, department name, policy version, or effective date), you can filter retrieval to return only chunks from documents matching those criteria. This is particularly useful when different user roles should see different document scopes.
The Rerank API is a separate capability that sits between retrieval and generation. If you retrieve 20 candidate chunks and want the generation model to focus on the most relevant five, a reranker scores the candidates and reorders them. This is covered in a separate article.
Failure modes
Wrong hit rate from the start. If you configure a chunking strategy without validating retrieval quality on a representative question set, you can ship a system that fails silently. The generation model will produce confident but unsupported answers when the correct passage is not in the retrieved set.
Chunk boundary splits answer-bearing content. Fixed-size chunking with low overlap commonly splits logical units. Policy documents, legal clauses, and specification documents with structured sections are especially vulnerable.
Semantic chunking ingest fails silently. Because semantic chunking uses a model at ingest time, model errors, rate limits, or document parsing failures at ingest can produce a knowledge base where some documents were chunked and embedded correctly and others were not. Check the data source sync log after each ingest run.
Embedding model mismatch after environment change. If you use different knowledge base resource ARNs for dev and production and configure them separately, it is possible to use a different embedding model in one environment. Retrieval evaluation results from dev do not transfer to production under a different model.
Parent chunk size too large. In hierarchical chunking, an oversized parent chunk consumes a large portion of the available context window and may push other retrieved chunks out of the prompt. Set parent chunk size relative to your model's context window and the number of results you expect to include.
Diagnostic signals
A retrieval evaluation test before go-live is the most direct diagnostic. If you do not have one, look for these signals in production:
- Generated answers that consistently miss exception clauses or qualifications suggest chunk boundaries are splitting those clauses from the main content.
- High answer refusal rate ("I don't have that information") combined with documents that should cover the topic suggests a retrieval hit rate problem.
- Answers that correctly cite one part of a policy but miss contradicting sections suggest retrieval is returning only the first matching chunk.
The Bedrock Knowledge Bases monitoring documentation describes CloudWatch metrics available for knowledge base operations.
What to do next
The next useful step is building a retrieval evaluation set of 15-20 representative questions before deploying to production. Run retrieval-only evaluation and identify the minimum numberOfResults that achieves acceptable hit rate. Document the chunking strategy, embedding model, and dimensions used. Keep that record alongside your knowledge base ARN, because you will need it if you ever need to rebuild.
For the production deployment checklist covering index build, retrieval test record, guardrail configuration, and monitoring setup, see the Bedrock RAG Pipeline Deployment playbook.
For the Ampity team's work on RAG systems and retrieval infrastructure, see the LLM and RAG Systems service page.
Primary references
- Amazon Bedrock Knowledge Bases overview, source checked 2026-11-05
- Create a knowledge base by connecting to a data source, source checked 2026-11-05
- How content chunking works for knowledge bases, source checked 2026-11-05
- Supported models and Regions for Amazon Bedrock Knowledge Bases
Related resources
Bedrock Embedding Migration: What Must Be Rebuilt?
Decide when a Bedrock embedding change needs new document vectors, a rebuilt index or a matched query route, with a worked compatibility example.
RAG Architecture Playbook for Production Knowledge Systems
Build a RAG system with authorized retrieval, source lineage, measured answer support, access-revocation and deletion controls, and rehearsed index recovery.
Why Retrieval Changes Break Previously Good AI Answers
Trace an answer regression through document revisions, chunk boundaries, ranking, permissions and context assembly before changing the model or prompt.