AI Project Readiness: Decide What Is Ready to Build

Decide whether an AI use case is ready to build with a business baseline, data rights, representative evaluation, risk boundaries, and an operating owner.

Decide what the system will be allowed to do

An AI demonstration answers a narrow question: can this configuration produce a useful result on the examples shown? A production decision needs different evidence. Who will rely on the result? Which mistakes matter? Can the organization lawfully use the inputs? Who handles an incorrect answer after launch?

There is no useful universal failure percentage for that decision. Projects differ in what they call a pilot, deployment, success, and failure. Start with a specific workflow and a decision record that someone can approve, narrow, or reject.

This guide covers readiness before committing to production delivery. It applies to predictive models and generative AI, while distinguishing their evaluation problems. It does not require every project to adopt a feature store, an agent framework, or a fixed period of shadow operation.

Write a business acceptance statement before choosing a model

Name the task, its current owner, and the smallest useful change. “Introduce an AI assistant” is not an acceptance statement. “Help support agents draft answers to documented product questions, without sending messages or changing accounts” is specific enough to investigate.

Measure the existing workflow on representative work. Include time spent searching, checking, correcting, and escalating, not just time spent typing. If quality is the main problem, record the types of incorrect decisions and their consequences. A faster first draft may increase total work if checking it takes longer.

Compare the proposed system with a credible alternative. That might be improved search, a rules-based workflow, a smaller predictive model, or better documentation. Do not build a weak baseline merely to make AI look useful.

Use the enterprise AI readiness checklist to collect the supporting artifacts with the business, data, security and operating owners.

These fields are a working brief, not a compliance certificate. A promising model cannot compensate for an unresolved permission or an owner who cannot support the workflow.

Establish data rights and availability at the point of use

A dataset being accessible to a developer does not establish a right to use it for training, retrieval, or external inference. Check those purposes separately. Contracts, privacy obligations, internal classification rules, and retention requirements can impose different conditions on each.

For predictive models, confirm that every feature is available when the prediction must be made. A historical field entered after a case was resolved can produce an impressive retrospective result and be unusable at decision time. Evaluate with time-aware splits when the task depends on temporal ordering, and prevent related entities or duplicates from leaking across splits where that would inflate performance.

For retrieval-based generative AI, test authorization in retrieval itself. A correct answer drawn from another customer's private document is still a serious failure. Preserve document ownership and access changes through ingestion, indexing, retrieval, and any answer cache. Removing a source from the main repository is not enough if a stale index or retained prompt still exposes it.

Also ask whether the sources support the intended answer. Policies can conflict; documents can be obsolete; a corpus can omit an important customer segment. Retrieval does not turn missing or contradictory evidence into ground truth. The narrower data-readiness guide for generative AI covers that work in more detail.

Build an evaluation that can reject the idea

Reserve evaluation examples before repeated prompt or model tuning. Document how they were sampled and have appropriate reviewers assess them against a written rubric. Keep a record of disagreements rather than silently turning subjective judgments into an objective score.

A representative set should include normal work, costly mistakes, unusual but legitimate requests, and cases where the system should abstain. Report results by consequential slice. An overall average can hide failure on a less common language, product, or customer group.

For a predictive model, choose measures that match the decision. Precision and recall have different consequences when false alarms create review work and missed cases create harm. Calibrated risk estimates matter if downstream users interpret a score as a probability. Delayed outcomes may prevent a final quality judgment at release time; record that limitation.

For generative AI, separate answer correctness, support from authorized sources, usefulness, abstention, and unsafe actions. A plausible citation is not proof that the cited document supports the answer. Review the cited passage and the claim together. Include malicious instructions in retrieved text and user inputs, but do not treat passing a test set as proof that prompt injection has been eliminated.

The NIST Generative AI Profile provides a risk-management reference for this additional work. Its existence does not validate a particular model or deployment.

Worked decision brief: a support-drafting assistant

The following is a hypothetical planning example, not an Ampity customer result.

A support team wants drafts for documented product-configuration questions. It excludes refunds, account changes, security incidents, and customer-specific contract interpretations. The assistant can retrieve approved product documentation and propose a draft. It cannot send a response or execute a tool that changes customer state.

The team compares three workflows on a held-out collection of de-identified, permitted cases: existing search, improved search, and retrieval-assisted drafting. Reviewers see the question, the relevant authorized evidence, and the proposed response. They record unsupported claims, incorrect instructions, missed escalation, and the work needed to make the response usable.

The acceptance criteria belong to the support owner and risk reviewers. Set them before comparing candidates, including which failures block release regardless of average performance. A small test with no observed critical failures does not prove the real-world failure rate is zero.

An initial deployment, if approved, remains a drafting aid with human review. That review is a control only if reviewers have time, evidence, training, and authority to reject the output. It is not a reason to send unreliable drafts into a high-volume workflow.

Treat missing labels as uncertainty, not evidence of safety

Fraud screening illustrates a different readiness problem. If a previous system blocked a transaction, the organization may not observe what would have happened had it proceeded. Historical labels are partly shaped by the earlier decision policy. This is selective-label or feedback bias, not automatically a cold-start problem or label leakage.

Do not mark blocked transactions as confirmed fraud, or as confirmed legitimate, simply to complete a training table. Human review can add evidence, but it does not necessarily reveal the unobserved counterfactual.

Nor is there a general safe percentage of risky transactions to allow through for learning. Any live experiment that changes financial exposure needs a qualified risk owner, explicit authority, exposure limits, monitoring, and a stop procedure. Offline analysis and shadow scoring can support a decision, but neither alone demonstrates the outcomes of a policy that changes real actions.

This distinction matters outside fraud too. Recommendations affect what users see; triage affects which cases receive attention. A new system can change the data used to evaluate the next version. Record that feedback path in the readiness brief.

Budget for a maintained system, not a model endpoint

Google's original Hidden Technical Debt in Machine Learning Systems paper describes system-level maintenance risks such as data dependencies, feedback loops, and entanglement. It does not justify assigning a universal percentage of project work to model code.

Estimate the costs your workflow actually creates: inference, retrieval, data refresh, evaluation, human review, incident response, and integration maintenance. Include latency and provider-limit behavior. Test a provider outage or quota failure with the intended fallback, rather than assuming users can simply revert to the old workflow.

Record the deployable configuration: model identifier, prompt and policy versions, retrieval configuration, source snapshot or index version, evaluation set, and release decision. Hosted model providers may not expose training data or guarantee identical outputs across runs. Document that limit instead of promising complete reproducibility.

A change to a prompt, tool permission, data source, or retrieval filter can change system behavior without a model update. Give those changes a release path. Drift or user complaints should trigger investigation; they are not automatic instructions to retrain and redeploy.

Detailed model release and monitoring mechanics belong in the production MLOps guide. Policy ownership and approval responsibilities belong in the AI governance guide.

Make the go, narrow, or stop decision explicit

Choose go when the evidence supports the defined scope and an accountable owner accepts the residual risk. Choose narrow when a smaller task can be evaluated and operated safely. Choose stop when the required permission, evidence, control, or business value cannot be established at a reasonable cost.

Before approving the next stage, ask:

• Is the task still useful without assuming perfect model behavior?
• Can each source be used for the intended purpose, and are access boundaries enforced?
• Can the evaluation expose the failures that matter, including abstention and escalation?
• Are total workflow cost and reviewer capacity understood?
• Can the operating team detect a problem, stop the affected behavior, and preserve evidence?
• Is the decision recorded with unresolved questions, owners, and conditions for reconsideration?

The NIST AI Risk Management Framework is a useful reference for organizing risk work across governance, context, measurement, and management. The practical output here is smaller: an evidence-backed decision about one workflow.

Bring that brief, a representative permitted sample, and the unresolved constraints to an AI readiness and feasibility assessment. If the main uncertainty is rights, coverage, or source quality, start with a data-readiness assessment. Neither requires pretending that a successful demo has already proved production value.