Vector Database Selection: Benchmark, Authorize and Migrate

Select a vector store using filtered retrieval quality, workload cost, tenant controls and recovery tests, then migrate with durable changes and reconciliation.

trigger="A retrieval workload needs a new vector store, or measured constraints justify changing the existing one." owner="The retrieval/platform lead owns selection; the data owner accepts synchronization and recovery, and security approves the access boundary." participants={["Retrieval engineer", "Platform/database operator", "Application owner", "Security/data owner", "Finance/procurement partner"]} prerequisites={["An authorized corpus and query evaluation set", "A versioned embedding and chunking recipe", "Tenant, document and deletion requirements", "Workload scenarios, service objectives and operating capacity"]} outputs={["A reproducible candidate scorecard", "A dated workload-based cost model", "Authorization and recovery test evidence", "A migration plan with reconciliation and a safe read rollback"]} doneWhen={["The candidate meets mandatory quality and security requirements", "Cost and performance were measured under representative conditions", "Updates, deletes and permission changes propagate correctly", "Cutover and fallback preserve the authoritative data and access rules"]} />

Choose the retrieval behavior, not a vendor ranking

Vector count alone does not decide which database to use. Query filters, embedding dimensions, update rates, tenant skew, concurrency, availability and the team's operating capacity all affect the choice.

This playbook evaluates a vector store and its migration. The RAG architecture playbook covers the wider application, including grounding, generation and evaluation. A good vector benchmark does not establish that generated answers are correct or authorized.

Do not start from an undated price or a label such as “fastest.” Keep the current store as a candidate. Extending an existing database may be sensible, but only if the added retrieval workload does not compromise transactional objectives. A separate system may isolate load while adding synchronization and operational work.

1. Write the workload and mandatory requirements

The retrieval owner defines the corpus, query population and required behavior. The platform owner adds failure and maintenance conditions; security defines access boundaries.

Vector workload card
  Corpus owner, classification and permitted regions:
  Documents, chunks, vectors and projected growth:
  Embedding model/version, dimensions and distance metric:
  Query and document encoding recipes:
  Query classes, filters, tenant sizes and skew:
  Concurrency, arrival pattern and response objectives:
  Insert, update, delete and permission-change rates:
  Freshness and deletion-propagation requirements:
  Recovery objectives and source-of-truth location:
  Operating owner, candidate version/edition and test date:

Use compatible query and document encoders according to the chosen model's instructions. Some retrieval models use distinct prefixes or encoding paths. Matching dimensions alone does not make vectors comparable. Keep model, normalization, distance metric and preprocessing versions explicit.

Separate mandatory requirements from preferences. An option that cannot meet data-location, tenant-isolation or recovery requirements should not win because it has a better average latency score.

Gate: the candidate test can be reproduced from the workload card, and each requirement has an acceptance owner.

2. Shortlist by operating model and verified capability

Compare an extension to an existing data platform, a self-operated dedicated engine and a managed service where they are feasible. Name the exact edition, version and deployment configuration. A product's managed offering may differ from its open-source distribution.

| Option shape | Potential reason to evaluate it | Question that can disqualify it | |---|---|---| | Existing relational platform with vector support | Shared operational skills and nearby transactional data | Does indexing or retrieval interfere with critical database work? | | Dedicated engine operated by the team | Required control over deployment and tuning | Can the team patch, scale, restore and monitor it? | | Managed retrieval service | Delegate some infrastructure operation | Do region, identity, recovery, cost and contract terms fit? |

Use current official documentation for candidate-specific behavior. For example, pgvector documents its index and filtering behavior, Qdrant documents filter semantics, and Weaviate documents filtering in its search path. These are implementation references, not evidence that one wins your workload.

No entry-level price or free-tier allowance is reproduced here. Record dated quotes and billing units for the selected edition when evaluating it. A feature absent from an old comparison may now exist; a feature available in one edition may not be included in another.

3. Build two quality baselines

The retrieval engineer separates nearest-neighbor accuracy from task relevance.

First, compare approximate search with an exact-search baseline over the same authorized corpus, embeddings, metric and filters. Recall at k measures overlap with the exact neighbor set, using a defined denominator and tie policy. If a filtered corpus contains fewer than k eligible records, document how the metric handles it.

Second, evaluate whether the retrieved material actually supports the user's task. Use domain-labeled queries, expected relevant records and hard negatives. Include ambiguous questions, rare terminology, different languages, empty results and documents that should never be visible.

An exact nearest neighbor can still be irrelevant. A high ANN recall score does not prove useful retrieval, and a broad embedding leaderboard does not replace domain evaluation. Keep a held-out set so index and model tuning do not merely optimize the same examples repeatedly.

Gate: product/domain reviewers accept the task-quality criteria before the candidates are ranked.

4. Benchmark filtering, writes and contention together

Hold the embedding and chunking recipe constant when comparing stores. If you also change the model, run a separate experiment so the cause of quality differences remains understandable.

| Test slice | Measure | Failure that the average can hide | |---|---|---| | Broad and selective filters | Eligible result count, recall and latency | Highly selective tenant filters return too few candidates | | Small and large tenants | Per-tenant quality, latency and resource use | A large tenant degrades others | | Concurrent ingestion and search | Freshness, throughput, errors and latency distribution | Index builds or updates stall reads | | Updates, deletes and revocation | Time until old content is no longer retrievable | Successful writes are acknowledged before search catches up | | Restart, restore and maintenance | Recovery time, missing records and consistency | A benchmark works only with warm caches | | Hybrid retrieval or reranking | Task quality and end-to-end cost/latency | A faster vector stage does not improve the full request |

Tune one factor at a time within resource limits. HNSW, IVF-style indexes, compression and exact scans make different quality/resource tradeoffs; there is no universal recall guarantee or parameter setting. Increasing a search budget may recover candidates at additional cost.

In pgvector, approximate-index filtering can return fewer results; the project documents iterative scans and their version-dependent limits. This is a reason to test filtered queries, not to claim all pre-filtering requires a full scan or all post-filtering is faster.

Gate: report distributions and worst-performing relevant slices, not only a single mean or an unfiltered vendor benchmark.

5. Enforce authorization outside user-supplied filters

The application derives tenant and principal identity from authenticated context. It resolves permitted document scope using trusted policy, then constructs the database query. A user, model or retrieved document must not supply or widen that scope.

"type": "flow", "title": "Retrieval must preserve the caller's authorized scope", "steps": [ ], "caption": "Security policy is independent of the model's query. Recheck current access before returning content to a user or an external generation service." }} />

Use database enforcement where appropriate as defense in depth. PostgreSQL's row-security documentation explains policies and important bypass conditions, including superusers and roles with BYPASSRLS; table owners normally bypass policies unless forced. Test the actual application role, not an administrator connection.

A shared connection-pool context must be set and cleared safely for each request. Cache keys and caches must also preserve access scope and revocation behavior. Do not treat a hardcoded tenant filter or parameterization alone as authorization.

Test cross-tenant reads, forged tenant IDs, revoked access, deleted documents and stale replicas. For sensitive content, a stale index must not bypass a current permission decision. If the policy service is unavailable, use an explicitly approved restricted behavior rather than widening access.

6. Model the cost of an accepted workload

The finance/platform owner records region, currency, quote date, edition, usage basis and committed minimums. Pinecone's cost documentation is an example of why the selected service's billing model must be inspected rather than inferred from vector count.

Include ingestion, search, storage, replicas, backups, network transfer, support and operations. Track embedding, reranking and rebuild costs separately so a database comparison does not hide them. Include temporary duplicate storage and processing during migration.

Candidate cost record
  Version/edition, region, date and official quote/reference:
  Workload scenario and accepted quality level:
  Read/write/storage or infrastructure units:
  Replicas, backups, transfer and support:
  Engineering and operating effort:
  Re-embedding, rebuilding and migration overlap:
  Monthly total by scenario:
  Cost per accepted query, with denominator definition:
  Unpriced limits, minimums and exit charges:

Compare options at an accepted quality and freshness level. A cheaper configuration that fails mandatory retrieval or deletion requirements is not a comparable result. Avoid extrapolating linearly through capacity or billing boundaries without testing them.

7. Migrate from an authoritative source, not casual dual writes

The data owner identifies the authoritative documents and a durable sequence of changes. A pair of application writes to two databases is not atomic. One can fail, time out after success, arrive out of order or miss a delete.

Use a supported change-capture mechanism or a transactional outbox where applicable. AWS's outbox guidance explains the database/publication boundary. Consumers still need idempotency, ordering rules and reconciliation.

Capture a consistent snapshot with a matching change position. Backfill stable document/chunk identifiers, record source versions and retain tombstones or equivalent delete state. Replay changes from the correct position. Ensure a late backfill or retry cannot overwrite a newer update or resurrect a deleted document.

"type": "flow", "title": "A new index earns traffic through reconciliation", "steps": [ ], "caption": "The authoritative source and durable change stream remain available through cutover. Returning reads to the old index is safe only while its data and permissions remain current." }} />

Reconcile identifiers, versions, content or vector checksums where meaningful, permissions and deletion state. Counts alone cannot distinguish one missing record from one duplicate or an update from stale content. Keep replay checkpoints and a repair queue with an owner.

Search visibility may lag write acknowledgement. Pinecone's freshness documentation describes eventual consistency and version-specific mechanisms to check indexing progress. Whatever the store, test the actual read-after-write and delete behavior required by the application.

Gate: no read cutover until the backlog, reconciliation exceptions, quality tests and access tests meet the agreed criteria.

8. Cut over with an honest recovery boundary

The release owner moves a bounded cohort and compares quality, errors, latency, freshness and rejected-access cases. Pick exposure and observation windows from workload and risk. Do not retire the old index after an arbitrary number of days.

| Failure | Immediate action | Recovery condition | |---|---|---| | New index misses or exposes records | Halt cutover and restrict affected retrieval | Reconcile content and permissions before resuming | | Change consumer falls behind | Hold traffic progression and preserve change retention | Replay safely without overwriting newer state | | Model/version mismatch | Stop affected queries | Route to a compatible index and rebuild the incorrect generation | | New store becomes unavailable | Use the approved fallback or return a bounded failure | Confirm fallback freshness, capacity and access behavior | | Old index is stale during rollback | Do not blindly return traffic | Catch it up or rebuild from the source before reuse |

A model migration usually needs a separately versioned vector generation and corresponding query encoder. Do not mix incompatible spaces because their dimensions happen to match. Keep the old recipe available for the fallback window.

Retire the old index only when consumers, data checks, recovery obligations and retention decisions are closed. Securely remove obsolete credentials and data under the approved policy.

9. Adjudicate retrieval quality with domain evidence

Create an evaluation set from real information needs, difficult filters, ambiguous terms, access boundaries and recently changed content. Keep a held-out portion that was not used to tune chunking, embeddings, index parameters or reranking. A benchmark repeatedly tuned against the same questions becomes development data and can overstate expected quality.

For each question, record the eligible source set, relevant evidence, unacceptable evidence, required freshness and whether abstention is preferable to a weak result. Domain reviewers should judge retrieved evidence before judging a generated answer. This separates a retrieval failure from a model synthesis failure.

Report more than an overall relevance average. Break results down by tenant size, language, document type, filter selectivity, update age and other consequential slices. Record queries with no known relevant result. A system should not be rewarded for returning plausible but unauthorized or nonexistent evidence.

When reviewers disagree, preserve the disagreement and adjudication rule. The problem may be an unclear task, conflicting source documents or a policy decision rather than a database defect. Update the evaluation definition before using that case as a release gate.

Accept a candidate only when mandatory access, deletion and freshness conditions hold and the quality tradeoff is understood. Faster search does not compensate for leakage. Higher recall does not justify returning evidence outside the user's permitted scope.

10. Plan day-two operations before vendor selection

The operating owner documents how the team will observe ingestion lag, query failures, latency distributions, capacity, index health, backups or exports, and authorization mismatches. Define which signals create an alert, who can act, and what safe degraded behavior exists. A dashboard without response authority is not an operating plan.

Exercise one maintenance event and one recovery event with the candidate. Maintenance may involve index rebuild, schema or metadata change, software upgrade, capacity expansion or credential rotation. Recovery may involve restoring from an export, rebuilding from the authoritative source, replaying changes and proving permissions and deletion state. Measure the complete time to accepted retrieval, not only the time to create an empty index.

Check exit behavior. Export a representative dataset with stable identifiers, metadata, permissions, source versions and deletion state. Recreate it through a second path and identify vendor-specific features that require redesign. Record transfer cost, rate limits, support dependency and the period of duplicate infrastructure. Portability is demonstrated by an exercise, not by using a nominally common vector format.

Assign lifecycle ownership for the embedding model, chunking recipe, reranker, database version and client library. A change to any of them can invalidate comparisons or require a new vector generation. Keep compatibility metadata with each index so an operator cannot route a query encoder to an incompatible vector space.

Review the selection when corpus composition, access policy, workload shape, freshness requirement, provider terms or operating capacity changes materially. The chosen system is a fit for a documented scenario, not a permanent winner.

Keep a dated decision record with the rejected alternatives and their evidence. A future team should be able to see whether a candidate failed a mandatory requirement, lost on measured quality, exceeded an operating constraint, or was excluded because a required capability was not verified. This prevents an old product ranking from being treated as current truth and makes a later re-evaluation cheaper.

Recheck the entire shortlist before procurement or production approval, because editions, limits, regions and support terms can change without preserving earlier assumptions.

Selection and migration acceptance checklist

"The workload card includes filters, tenant skew, updates and recovery", "Candidates are identified by version, edition, region and date", "ANN recall and domain relevance are evaluated separately", "Authorization is derived from trusted context and independently enforced", "Costs use current official billing inputs and an accepted workload", "Backfill is paired with durable, ordered change handling", "Deletes, access revocation and stale retries cannot resurrect content", "Cutover evidence includes reconciliation rather than counts alone", "The fallback remains current until its retirement is authorized" ]} />

Results apply only to the tested corpus, workload and versions. This playbook does not certify every vendor feature or guarantee answer quality. Start by collecting representative filtered queries and a current access policy; those inputs are more useful than a generic ranking. LLM and RAG systems is the relevant service scope for implementing the retrieval boundary.