Enterprise AI Readiness Checklist: Evidence Before a Pilot

Test whether an enterprise AI use case has a decision owner, usable data, measurable quality gates, safe integration and an operating path before funding a pilot.

Readiness is evidence that a bounded use case can be operated

An enterprise is not ready for AI because it has a model account, a data lake or an executive sponsor. Readiness means a team can name one decision or workflow, obtain the permitted inputs, evaluate the output against an accepted baseline, integrate the system safely and operate it when the model or a dependency behaves differently than expected.

Use this checklist before funding a pilot. It is designed for a business owner, product lead, engineering lead, data owner, security representative and operations owner working together. A positive answer needs an artifact or observed result. “We should be able to” is an open item.

The checklist does not produce a universal readiness score. A missing requirement can block one use case while being irrelevant to another. Rank gaps by their effect on the proposed workflow and the consequence of a wrong action.

1. Define the decision and the human authority

Write the use case as a change to a real workflow. Name the person who acts today, the input they receive, the decision or task they perform, the output produced and who is affected when it is wrong.

“Use generative AI for customer support” is too broad. “Draft a response for a support agent using approved product documentation, with the agent retaining send authority” defines an actor, source boundary, output and approval point. It can be evaluated.

Record which actions remain human decisions. If the system can call tools, change records, issue refunds, send messages or alter access, list each action and its authorization rule. A chat interface does not make an action low risk.

Evidence to require:

• named business owner and technical owner;
• current workflow and baseline;
• included and excluded user groups;
• permitted actions and escalation path;
• harm or loss if the output is wrong, late, missing or disclosed.

AWS's current Responsible AI Lens recommends narrowly defined use cases because the use case determines risk and release criteria. See Responsible AI design principles.

2. Establish a baseline before choosing a model

Measure the current workflow. Depending on the use case, this may include completion time, review effort, error categories, abandonment, escalation, cost per completed task or customer outcome. Define the denominator and observation period.

The baseline can be imperfect. It must be explicit enough to compare the proposed system with the current approach. A pilot that reports “80% accuracy” without defining the task, sample, error severity or baseline cannot support a release decision.

Create a small evaluation set from representative historical or constructed cases. Include ordinary examples, difficult cases, missing information, conflicting instructions and cases that should be refused or escalated. Keep the set versioned. Separate examples used during development from a holdout set used for release review.

NIST's Generative AI Profile organizes risk work through govern, map, measure and manage. It emphasizes context-specific risk management rather than one score that applies to every system. See NIST AI 600-1.

3. Verify data rights, quality and retrieval boundaries

Inventory every data source the system will use. Record its owner, purpose, sensitivity, retention, geographic or contractual restrictions, update frequency and deletion process. Include prompts, retrieved context, feedback, traces and outputs. These are part of the data system even when they live in different products.

For retrieval-augmented generation, test whether the retrieval layer enforces the user's authorization before documents enter the prompt. Filtering the final answer is too late if another tenant's or another employee's content has already been exposed to the model or captured in traces.

Check content quality at the unit the model retrieves. A well-governed document repository can still produce poor retrieval if chunks lose titles, dates, product versions or permission context. Measure retrieval separately from answer quality so the team knows whether a failure came from search, context assembly or generation.

"type": "svg-flow", "title": "Evidence path from governed source to evaluated answer", "steps": [ ] }} />

4. Turn quality into release criteria

Choose evaluation dimensions that match the workflow. A document-grounded answer may need retrieval recall, citation support, factual consistency, refusal behavior and reviewer acceptance. A classifier may need class-specific precision and recall. An agent may need task completion, correct tool choice, argument validity, policy compliance and safe termination.

Do not average a severe failure into a healthy score. Separate blocking failures such as unauthorized disclosure or an unapproved external action from quality problems that can be reviewed. Set thresholds from business tolerance and the baseline, not from a generic benchmark.

Run deterministic checks where possible. Schema validation, permission checks, tool allowlists, amount limits and duplicate-action protection should not depend on another model's opinion. Use human review for judgments that require context, and document reviewer guidance and disagreement handling.

Evaluate each material change to the model, prompt, retrieval pipeline, tool contract or policy. A model name alone is not the system version. The OpenAI Evals API, for example, represents an evaluation as testing criteria plus a data-source schema and supports runs against model and parameter choices. The useful principle is versioned tests and comparable results, regardless of provider. See the OpenAI Evals API.

5. Design the integration and fallback

Draw the complete path from user or event to business system. Include identity, prompt and configuration storage, model provider, retrieval, tools, queues, rate limits, output handling, audit records and human escalation.

For every dependency, decide what happens when it is slow, unavailable, rate-limited or returns an unexpected response. A fallback can be the current manual workflow, a reduced capability, queued work or a clear refusal. Silently switching to a different model can change cost, latency, regional processing and output behavior, so treat it as a governed release path.

If the system writes to a business system, use stable operation identity and verify the result before retrying. A timeout does not prove an action failed. The model must not create a new payment, ticket or message merely because the previous call's acknowledgement was lost.

6. Confirm security and privacy controls at the AI boundary

Threat-model the actual application, not only the model endpoint. Test direct and indirect prompt injection, excessive tool authority, sensitive information disclosure, poisoned source content, insecure output handling and cross-tenant retrieval.

Keep untrusted content distinct from instructions. Restrict tools by user and workflow context. Validate arguments in the application. Apply least privilege to model, retrieval and tool identities. Log decisions without placing secrets or unrestricted personal data into traces.

NIST AI 600-1 identifies risks that generative AI can introduce or intensify and provides actions across the lifecycle. AWS's Generative AI Lens evaluates scoping, model selection, customization, development, deployment and continuous improvement against operational, security, reliability, performance, cost and sustainability concerns. See the Generative AI lifecycle.

7. Build an operating model before the pilot ends

Name the owner for prompts, evaluation data, retrieval content, model configuration, security response, cost and user support. Decide who can release a change and who can disable the capability.

Monitor user-visible task success, blocking safety failures, latency, dependency errors, token or model cost, retrieval quality, escalation and feedback. Sample traces according to risk and privacy constraints. Alert on conditions that require action, not every variation in model output.

Set a review schedule for provider changes, new use cases, evaluation drift and source-data changes. A pilot can rely on a small expert group. A production system needs support, access review, incident handling, recovery, evidence retention and a budget owner.

"type": "svg-comparison", "title": "Prototype evidence versus production evidence", "headers": ["Prototype question", "Production gate"], "rows": [ ] }} />

8. Make a bounded readiness decision

Choose one of four decisions:

| Decision | When it fits | Required next action | | --- | --- | --- | | Proceed to controlled pilot | The task, data boundary, evaluation and owner are defined; remaining uncertainty can be tested safely | Approve pilot scope, participants, data and exit criteria | | Proceed after prerequisites | The use case is viable but a blocking dependency is known | Close the named gap and re-run the affected checks | | Redesign the use case | The proposed autonomy, data need or consequence is too broad | Narrow the task, authority or user population | | Stop | The workflow lacks value, lawful or contractual data use, measurable quality or an accountable owner | Record the reason and avoid a technology-only pilot |

Readiness is use-case specific. An organization may be ready for internal drafting with human approval and not ready for automated customer decisions. Preserve that distinction instead of reporting one enterprise-wide percentage.

Failure patterns that invalidate a readiness score

A high average score is misleading when the assessment hides one of these conditions:

• the use case is described as a capability such as “enterprise copilot” instead of a bounded task and decision;
• evaluation examples were selected after seeing the system’s outputs;
• the same people who built the prototype judge every ambiguous result without reviewer guidance;
• source rights, user permissions or deletion obligations are assumed rather than traced through derived stores;
• the pilot depends on manual interventions that are absent from the proposed production operating model;
• the model or tool can take an irreversible action without an application-enforced authority check;
• a provider, retrieval or business-system failure has no tested fallback;
• projected cost excludes evaluation, observability, human review, support and exception handling.

These are not minor maturity gaps. They can change the decision from “pilot” to “redesign” because the proposed evidence does not represent the system that would actually operate. The practical tradeoff is scope: narrowing the audience, data set or action authority often produces better evidence faster than adding more model capability.

Evidence checklist

"One workflow, user population, decision owner and excluded scope are written down.", "The current workflow has a defined baseline and observation period.", "Data ownership, permitted use, access, retention and deletion are recorded.", "A versioned evaluation set covers ordinary, difficult, refusal and escalation cases.", "Blocking safety failures are separated from average quality measures.", "The integration handles timeout, retry, provider failure and duplicate actions safely.", "Security testing covers untrusted instructions, tool authority and information disclosure.", "Release, support, cost, incident and disable owners are named.", "The pilot has explicit entry, exit and stop criteria." ]} />

If any blocking item is open, do not conceal it inside a readiness score. Record the missing evidence, owner and decision date. Start an AI readiness and feasibility assessment when the use case is still broad, or a data readiness assessment when source quality, permissions or retrieval are the main uncertainty.