Data Readiness for Generative AI: An Evidence Checklist

Assess whether an information corpus is safe and useful enough for a generative AI system across rights, access, quality, retrieval, freshness, deletion and evaluation.

Data readiness is an acceptance decision

A folder of documents is not a production-ready AI knowledge base. The team needs evidence that it may use the material, can preserve access rules, can retrieve the right evidence and can remove information when the source changes.

The decision is specific to one use case. Data that works for an internal drafting assistant may not be appropriate for a customer-facing answer system.

The seven checks

"type": "svg-flow", "title": "Generative AI data-readiness checks", "steps": [ ] }} />

1. Rights and purpose

For each source, record the owner, license or agreement, intended audience, geographic or contractual limits, retention requirement and whether derived use for AI is permitted. Public availability does not automatically grant unrestricted reuse.

Keep legal interpretation with the organization’s qualified advisers. The engineering record should show which approval the system relies on and what technical controls implement it.

2. Access propagation

An answer system must not reveal a document to someone who could not read the source. Preserve the relevant access attributes during ingestion and enforce them at retrieval time.

Test mixed-permission questions where an authorized source and a restricted source contain similar language. The result must use only evidence available to that user. Apply the same rule to previews, caches, conversation history and evaluation logs.

3. Content quality

Measure what affects the use case:

  • Authority: which source wins when two documents disagree?
  • Coverage: do the documents answer the questions the system is expected to handle?
  • Structure: can the ingestion process preserve headings, tables, lists and relationships?
  • Duplication: are there several versions that look equally current?
  • Identifiers: can the system connect products, policies, customers or records consistently?
  • Sensitivity: does the content contain personal, confidential or regulated information?

Do not hide quality problems behind a larger context window. Conflicting and obsolete content remains conflicting and obsolete after embedding.

4. Retrieval fitness

Create a representative question set before selecting chunk sizes or a vector database. For each question, record the evidence a competent person would need and the documents that contain it.

Measure whether retrieval returns that evidence at a useful rank. Include short factual questions, multi-document questions, vague terminology, acronyms, recent updates and questions that should receive no answer.

5. Freshness and versioning

Define what makes a document current, how quickly an update must appear and how superseded versions are marked. Use stable source identifiers so an updated document replaces or versions the correct derived records.

Monitor ingestion lag and failed updates. A system that answers from yesterday’s policy after today’s source changed has a data pipeline problem, not a prompt problem.

6. Deletion and retention

Deletion must cover the original ingestion copy, extracted text, embeddings, search indexes, caches and any stored prompts or responses governed by the same request. Record where each derivative exists and how the system confirms removal.

Backups and immutable logs may follow different retention rules. Make that difference visible instead of promising immediate deletion the architecture cannot perform.

7. Evaluation and release evidence

NIST’s Generative AI Profile treats risk management as a lifecycle activity aligned to the system’s goals, risk tolerance and context. For data readiness, that means the release decision should combine retrieval quality, answer quality, security, privacy and operating controls.

"type": "svg-architecture", "title": "Evidence-preserving RAG architecture", "nodes": [ ], "links": [ ] }} />

Identify failure modes before the pilot

A readiness assessment should make the unsafe paths visible, not only describe the preferred pipeline. Test at least these conditions:

  • the authoritative document is missing, duplicated or contradicted by an older version;
  • a user asks a question whose best evidence is outside their permission scope;
  • the ingestion job succeeds partially and leaves an older derived record active;
  • a document is deleted at the source but remains available through an index, cache or saved conversation;
  • retrieval finds relevant text but omits the clause that changes the answer;
  • a question has no sufficient evidence, yet the system produces a confident response;
  • an evaluation set reflects demo questions but not the language, ambiguity or access patterns of real users.

Decide the safe behavior for each case. Depending on the use case, that may be no answer, a request for clarification, a link to the authoritative source, a human review queue or a rollback to the previous corpus and retrieval configuration.

The main tradeoff is usually between coverage and control. Adding more sources may improve answer coverage while increasing permission complexity, conflicting versions and deletion obligations. Larger chunks can preserve context while lowering retrieval precision. Faster freshness targets reduce stale answers but increase pipeline and reconciliation demands. Record these tradeoffs instead of hiding them inside implementation settings.

Build a readiness record

Use a table with one row per source collection:

| Field | Evidence to record | |---|---| | Owner | Accountable business and technical owner | | Permitted use | Agreement, policy or documented decision | | Audience | Users and roles allowed to retrieve the content | | Update path | Source event, schedule and acceptable lag | | Deletion path | Derived stores and confirmation method | | Quality issues | Conflicts, gaps, obsolete versions and remediation owner | | Evaluation coverage | Questions, expected evidence and release threshold |

The result should be a decision, not a maturity score: ready for a bounded pilot, ready after named remediation, or not suitable for this use case.

Run a bounded acceptance exercise

Choose one audience, one information domain and a small set of decisions or tasks. Freeze a versioned source snapshot and run the representative question set against it. Review retrieval evidence separately from final-answer quality so a fluent answer cannot conceal weak grounding.

For each test, retain the user role, question, expected source, retrieved passages, final response, evaluator judgment and any policy decision. Include adversarial and negative tests alongside ordinary questions. A successful pilot should show where the system refuses or escalates, not only where it answers.

Release only when the team agrees on measurable thresholds and on the failures that are never averaged away. Permission leakage, untraceable evidence and an inability to remove governed content should remain blocking conditions even if aggregate answer quality is high.

Apply a release gate, not a confidence score

A corpus is ready for a bounded pilot only when the team can answer four practical questions with evidence:

  1. Can we use it? Rights, purpose, retention and accountable approval are recorded for every source collection.
  2. Can we contain it? Source permissions survive extraction, indexing, retrieval, caching, evaluation and deletion.
  3. Can we trust retrieval? A representative question set finds the required evidence, respects negative cases and exposes conflicting or obsolete sources.
  4. Can we operate it? Owners can observe ingestion lag, repair failures, reconcile derived stores, roll back a release and confirm deletion.

Record unresolved gaps as named remediation with an owner and acceptance test. Do not average a serious permission failure together with several healthy quality checks. A single unacceptable control can make the corpus unsuitable even when its overall score looks high.

For the pilot, version the source snapshot, ingestion logic, access policy, retrieval configuration, model and evaluation set. Keep the current system or a no-answer path available as the rollback. Re-run the gate whenever the source contract, audience, legal basis, retrieval method or answer behavior changes materially.

Related services: Data readiness assessment and LLM and RAG systems.

The next action

Inventory the source collections proposed for the first use case. Give each collection an owner, permitted-use decision, audience, update path and deletion path. Then write 25 to 50 representative questions with expected evidence. Any source or question without an accountable decision becomes remediation, not pilot scope.

Primary references: NIST AI Risk Management Framework, NIST Generative AI Profile, and OWASP Top 10 for Large Language Model Applications. Apply applicable privacy, copyright, contractual and sector-specific requirements with qualified advisers.