Data Platform Readiness for Production AI: A Decision and Evidence Standard

A practical standard for deciding whether a data platform can support production AI, covering authority, contracts, lineage, quality, privacy, evaluation datasets,...

audience="CTOs, CDOs, CIOs, heads of data and AI, platform and ML engineering leaders, product owners, security and privacy leaders, and operations teams deciding whether a data foundation can support a production AI use case." decision="Whether a specific AI decision can be built, evaluated, served, observed, corrected, and governed from the available data platform, and which platform, data-product, and operating gaps must close before release." position="Assess readiness one consequential decision at a time. Require named data authority, versioned contracts, reproducible lineage, decision-specific quality, privacy and access controls, representative evaluation evidence, serving objectives, outcome feedback, and an accountable operating model before granting production authority." scope="This paper is provider-neutral and applies to predictive, generative, retrieval, recommendation, and decision-support systems. It does not prescribe a lakehouse, warehouse, vector database, model, cloud, or universal maturity score." outputs={[ 'A bounded AI decision and consequence record', 'An authoritative data and contract map', 'A lineage and reproducibility evidence chain', 'A decision-specific quality and evaluation plan', 'A serving, feedback and drift operating model', 'A production readiness gate and remediation roadmap', ]} />

Executive summary

A data platform is not ready for AI merely because it stores a large volume of data or can train a model. Production readiness means that an organization can explain which data influenced a decision, whether that data was appropriate and current, how the system behaved for relevant groups and conditions, what happened after the decision, and how operators can contain or correct failure.

Readiness is specific to a use case. A weekly demand forecast, an internal document assistant, a fraud intervention, a youth-sports eligibility check, and an autonomous procurement action have different freshness, privacy, quality, traceability, evaluation, and recovery needs. A platform can be ready for one and unsafe for another.

This paper proposes a decision-first standard:

1. Define the decision, user, action, consequence, authority, latency, and human role before evaluating the platform.
2. Identify authoritative sources and owners. Preserve raw evidence, version data contracts, and make transformations and joins reproducible.
3. Measure quality against the decision and important slices, not a single global completeness score.
4. Separate training, evaluation, retrieval, serving, feedback, and audit datasets. Govern each according to purpose, privacy, retention, and access.
5. Build an evaluation set that represents expected, difficult, rare, and harmful conditions. Prevent leakage and preserve the exact versions used.
6. Operate freshness, features, retrieval, models, prompts, policies, and feedback as one versioned production path.
7. Observe data and outcome drift, delayed ground truth, cost, and failure. Give operators a contained response path.

The goal is not to delay every AI initiative until the entire enterprise data estate is perfect. The goal is to create one trustworthy end-to-end decision path, expose its gaps, and expand only after retained evidence shows that the operating model works.

1. Define the AI decision before the platform

Start with the decision or assistance the system will provide. Name the user, subject, input context, proposed output, allowed action, human role, consequence of error, detection delay, and recovery path.

“Build an AI platform” is not a decision boundary. “Recommend the next replenishment action for a planner, with a human accepting every order” is one. “Draft a support answer from approved product documents, without executing account changes” is another.

Use a decision record:

| Field | Required answer | | --- | --- | | decision | What prediction, recommendation, generation, classification, retrieval, or action is produced? | | subject and user | Who or what is affected, and who consumes the result? | | authority | Is the output advisory, approval-gated, or allowed to act? | | consequence | What can happen if the output is wrong, late, biased, leaked, unavailable, or unaffordable? | | horizon | Which history is relevant, and how quickly does the decision lose value? | | ground truth | Which later observation can confirm or challenge the result? | | recovery | Can an action be stopped, reversed, corrected, reconciled, or only explained? | | evidence | Which data, evaluation, policy, release, and outcome evidence determines acceptance? |

The NIST AI Risk Management Framework organizes AI risk activity around governing, mapping, measuring, and managing risk. Use those functions throughout the lifecycle. Do not treat the framework as a score that makes every use case equally safe.

2. Set the consequence and authority boundary

Data requirements become stricter as the system’s authority and consequence increase. A low-consequence internal summary can tolerate different uncertainty and latency from a system that denies a customer action, changes inventory, or initiates payment.

Classify consequence across safety, financial, legal, privacy, security, fairness, reputation, availability, and operational workload. Consider affected populations and asymmetric harm. A small global error rate can conceal severe behavior for a rare class or a high-value customer path.

Then limit authority. Prefer deterministic software where rules are stable. Use AI where ambiguity or scale creates value. Add approval when a person can meaningfully inspect the proposal. Keep the final decision human-owned when consequences are difficult to detect or reverse.

Define action limits such as allowed tools, spend, volume, recipients, environments, data classes, and time window. A model score should not silently become production authority. Deterministic policy should decide whether the output may be shown, queued for review, executed, or rejected.

3. Find the authoritative data, not merely available data

Inventory candidate sources, but identify which system owns each business fact. A copied table may be convenient for training yet stale, filtered, or semantically different from the operational source.

For each important field or document, record source owner, meaning, event time, processing time, history, correction behavior, delete behavior, classification, retention, quality expectation, access policy, and known limitations. Distinguish observed values from derived labels and inferred attributes.

Authority can change over time. A customer address may be authoritative in one service after an order and another before checkout. A forecast outcome may depend on later cancellation or fulfilment. Record temporal semantics instead of joining the latest snapshot to every historical event.

Avoid accepting a dataset because it is already in the warehouse. Confirm that the data was collected for an appropriate purpose and that its transformations preserve the meaning needed for the AI decision.

4. Create a contract and trust path

A data contract should identify producer, consumers, schema, semantics, event time, keys, classification, quality rules, freshness objective, change policy, and incident owner. It is a working agreement with enforceable checks, not a schema file alone.

Preserve source evidence and replay position before conformance. Validate schema, key integrity, range, referential relationships, timeliness, classification, and decision-specific invariants. Quarantine failures with reason and owner instead of silently dropping or repairing them in the trusted layer.

Conformance resolves domain meaning, identity, time, units, and history. Publication adds discoverability, access policy, service objective, version, owner, and consumer expectations. Only then should a data product be called trusted.

Contract evolution needs compatibility rules and a consumer migration window. A nullable field becoming required, a category changing meaning, or an event timestamp changing source can invalidate a model without breaking a pipeline.

5. Make lineage reproducible

Lineage should answer which source versions, code, configuration, joins, reference data, labels, exclusions, and policies produced a dataset or decision. Visual arrows without version and time are helpful documentation but weak evidence.

The W3C PROV overview defines a model for describing entities, activities, and agents involved in producing data. The OpenLineage specification provides an open event model for jobs, runs, and datasets. Either can inform a practical lineage implementation. Choose the detail needed for decisions, incidents, audits, and reproducibility.

Capture immutable dataset or snapshot identity, transformation revision, orchestration run, parameters, environment, source checkpoints, row or object counts, rejection counts, quality results, and publisher. For streaming paths, preserve event and schema versions plus offsets or checkpoints.

Test lineage by reproducing a historical training or evaluation slice. If the organization cannot reconstruct the data behind a model or explain an important output, the lineage is not yet sufficient for that decision.

6. Measure quality for the decision and its slices

Global quality scores hide the failures that matter. Define quality dimensions from the decision: completeness, validity, uniqueness, consistency, accuracy, timeliness, representativeness, label reliability, coverage, and stability.

Measure important slices by region, tenant, product, channel, device, language, age band where lawful and relevant, demand pattern, source system, new versus established entities, rare event, or another factor tied to consequence. Protect sensitive attributes and use them only under an appropriate purpose and access model.

The W3C Data Quality Vocabulary provides a vocabulary for expressing data quality metadata. A vocabulary helps teams publish comparable measures. It does not decide which thresholds make an AI decision safe.

Define failure behavior. Some quality failures should stop publication. Others can mark a feature unavailable, route to a fallback, reduce authority, warn a reviewer, or create an owned exception. Avoid filling missing values in a way that hides source-system failure.

Track quality through time. A dataset can pass at publication and become stale, skewed, or semantically invalid before use.

7. Preserve historical truth and prevent leakage

Training and evaluation must use information that would have been available at the decision time. Joining a current customer state, corrected outcome, or future aggregate to an old decision creates leakage and inflated performance.

Use point-in-time joins, effective dating, event time, source checkpoints, and feature or transformation versions. Preserve late-arrival and correction behavior. If the production system sees data two hours late, an evaluation built from perfectly backfilled history is misleading.

Separate entities across training, validation, and testing where repeated behavior can leak identity. Use temporal splits where conditions change over time. For forecasting, respect horizon and seasonality. For retrieval and generative systems, ensure evaluation questions and expected answers are not copied into the indexed corpus in a way that trivializes the test.

Document exclusions and survivorship. A model trained only on completed transactions may ignore abandoned or rejected journeys. A support assistant evaluated only on solved cases may miss ambiguous and policy-sensitive requests.

8. Govern privacy, consent, and purpose

Data that is technically accessible may not be appropriate for model training, retrieval, inference, monitoring, or human review. Record purpose and lawful basis or organizational authority with qualified privacy and legal input.

The NIST Privacy Framework offers a voluntary tool for identifying and managing privacy risk. Use it to consider data processing, individual impact, governance, communication, and control. Tailor the analysis to applicable law, contracts, users, and jurisdiction.

Minimize fields and history. Use aggregation, tokenization, pseudonymization, redaction, or synthetic data where those techniques reduce exposure without invalidating the decision. Do not call data anonymous without a re-identification assessment appropriate to the context.

Trace data subject requests and source deletions through derived datasets, indexes, feature stores, caches, prompts, logs, checkpoints, backups, evaluation sets, and model lifecycle decisions. State where deletion from a trained model is not technically straightforward and which retraining, exclusion, or risk treatment applies.

9. Separate access from appropriate use

Authorization controls who or what can access data. Appropriate use also asks why, for which decision, at which granularity, and with which downstream restrictions.

Use individual and workload identities, least privilege, environment separation, short-lived credentials, encryption, network controls, and auditable policy decisions. Restrict direct access to trusted layers and production stores. Separate platform administration, data publication, model release, and audit where consequence justifies it.

Attach purpose, data class, tenant, region, and approved consumer to access decisions where practical. A notebook user with broad query permission should not automatically have authority to export data into an external AI service.

Observe queries, exports, unusual volume, policy denials, sensitive-field access, service identities, and inactive privileges. Logs need protected retention and useful alerting. Retaining every event without owners or response paths does not create control.

10. Build an evaluation dataset as a governed product

An evaluation set is a release control. Give it an owner, version, purpose, scope, construction method, source lineage, access policy, review history, and change process.

Include representative everyday cases, important segments, rare conditions, boundary values, known incidents, adversarial inputs, missing or conflicting evidence, and high-consequence cases. Balance realistic prevalence with deliberate coverage of risks that are rare but important.

For predictive systems, define labels, observation windows, censoring, delayed outcomes, and disputed ground truth. For generative and retrieval systems, define source authority, answerability, required citations, refusal conditions, policy constraints, and grading rubrics. Include cases where the correct behavior is to ask, defer, or not act.

Prevent benchmark overfitting. Keep a protected acceptance set or rotate challenge sets. Record when evaluation items informed prompt, feature, model, or policy development. Human reviewers need calibration, conflict resolution, and examples of acceptable variation.

The NIST Generative AI Profile extends AI RMF considerations for generative AI risks. Use it as a source of risk prompts, not as a replacement for use-case-specific evaluation.

11. Connect features, retrieval, and serving to one versioned path

Training features and online features must use compatible definitions. Record code, source, time window, aggregation, null behavior, category mapping, and freshness. Test offline and online parity for important features.

Retrieval systems need corpus version, document authority, chunking, metadata, access filtering, embedding model, index configuration, update lag, deletion behavior, and citation mapping. A strong language model cannot compensate for an index that mixes tenants or serves superseded policy.

Serving should identify the model or endpoint, prompt and system instructions where used, feature or retrieval versions, policy, output schema, fallback, cache, and rollout cohort. Preserve enough context to reproduce or investigate an important result without logging unrestricted sensitive content.

Use typed interfaces between data, model, policy, and application layers. Validate ranges, units, missingness, schema, and output structure. Do not let a free-form output directly drive a consequential database or third-party action.

12. Define freshness and latency budgets end to end

Freshness is not one pipeline metric. The useful age of a decision includes source availability, ingestion, transformation, publication, feature or index refresh, cache, inference, and application use.

Set maximum age by input and decision. A profile attribute may remain useful for a day while inventory or entitlement may require seconds. State how the system behaves when one source is stale and another is current.

Measure event time and processing time. Detect stopped partitions, late sources, partial windows, and clock problems. A pipeline can finish successfully while processing yesterday’s incomplete data.

Allocate a latency budget across retrieval, features, model, policy, external tools, and application. Include tail latency and timeouts. A lower-quality fallback can be safer than a late answer only if the fallback’s consequence is understood.

13. Build feedback and ground truth deliberately

Production AI needs an outcome path. Record whether a recommendation was shown, accepted, changed, ignored, or blocked, then connect it to later business evidence when appropriate.

Do not treat every user click as ground truth. Users can accept a poor suggestion, reject a good one for reasons outside the model, or never observe the eventual outcome. Separate interaction feedback, expert review, operational outcome, customer correction, and policy incident.

Define the ground-truth delay. Fraud, churn, fulfilment, maintenance, and medical outcomes may take weeks or months. Maintain provisional and final labels with version and date. Recompute evaluation when material corrections arrive.

Protect feedback from manipulation and selection bias. Systems often receive labels only for investigated or acted-on cases. Use sampling, delayed review, counterfactual analysis where justified, and explicit uncertainty rather than presenting incomplete feedback as full truth.

14. Detect data, concept, and outcome drift

Data drift means input distributions changed. Concept drift means the relationship between inputs and outcome changed. Quality, label, policy, user, provider, or process changes can create both.

Monitor features, retrieval coverage, source mix, missingness, category novelty, label rate, confidence, refusal, human overrides, policy denials, error, latency, cost, and business outcome. Slice monitoring by important populations and operating conditions.

Set investigation thresholds from consequence and natural variability. A statistical alert is not automatically a production incident. Pair it with impact, sample evidence, and an operator decision.

Contain first when consequence is high. Reduce authority, route to review, freeze expansion, fall back, or disable a source while preserving evidence. Then determine whether the cause is data, model, prompt, policy, application, provider, or external change.

15. Make cost part of readiness

AI data cost includes source extraction, storage, transfer, transformation, quality checks, catalogue, lineage, feature computation, embedding, indexing, retrieval, training, evaluation, inference, observability, human review, retention, and reprocessing.

Attribute cost to use case, tenant, dataset, model, environment, and lifecycle stage where useful. Shared platforms need a visible allocation method. Otherwise a successful pilot can hide a production cost that grows faster than value.

Measure unit economics such as cost per evaluated record, useful retrieval, reviewed decision, accepted recommendation, or completed outcome. Include failure and retry. A cheaper model that creates more review or correction may cost more overall.

Set budgets and degradation policies. Lower context, smaller models, cached results, batch processing, tiered storage, or reduced retention can save cost, but each changes quality, latency, traceability, or recovery. Test the effect before treating it as an optimization.

16. Choose platform options from the decision workload

Several platform patterns can support production AI. The correct choice depends on the decision workload, existing estate, skills, ownership, latency, consistency, portability, and operating capacity.

A warehouse can be strong for governed structured data, stable SQL transformations, scheduled features, and familiar access controls. It can become awkward when teams need large unstructured corpora, multi-engine processing, fine-grained file reuse, or experimental workloads that do not fit its cost and performance model.

A lakehouse can provide open or reusable storage, diverse data forms, scalable processing, historical versions, and a shared foundation for analytics and ML. It still needs semantic ownership, publication discipline, serving paths, and workload management. Moving files into object storage does not create data products.

A feature platform can improve reuse and online consistency for predictive systems. It adds definitions, materialization, serving, monitoring, and another control surface. It is valuable when several production decisions reuse stable features or require low-latency serving. It may be unnecessary for one scheduled model with a simple transformation path.

A vector database or search engine can support retrieval over approved text, images, or other embeddings. It introduces chunking, embedding, metadata, authorization, deletion, freshness, and retrieval-evaluation decisions. It should not become the authoritative store for documents or permissions.

Streaming infrastructure supports fresh events and rapid response. It also adds ordering, duplication, replay, schema, late-arrival, checkpoint, and operating complexity. Batch is often safer and cheaper where the decision horizon permits it.

Managed services reduce infrastructure work and can provide mature security or reliability controls. They create provider, pricing, region, data, portability, and service-limit dependencies. Self-managed platforms offer more control but require real capacity for patching, scaling, backup, restore, security, and on-call operation.

Use the smallest combination that satisfies the decision’s evidence and service needs. Keep authoritative data separate from specialized serving representations. Preserve contracts and lineage across the boundary so a feature store, index, cache, or model endpoint cannot quietly become an unexplained source of truth.

17. Evaluate options and trade-offs explicitly

Readiness decisions involve trade-offs. Record them rather than presenting one architecture as universally best.

| Option | Benefit | Trade-off to test | | --- | --- | --- | | centralized platform | common controls, metadata and reuse | central backlog, weak domain meaning, blast radius | | federated data products | domain ownership and scalability | inconsistent practices, duplicated capability, harder discovery | | real-time processing | fresher decisions and faster feedback | cost, ordering, replay and operational complexity | | batch processing | simpler recovery and predictable cost | stale decisions and delayed detection | | shared feature or retrieval service | reuse and consistent serving | coupling, noisy neighbors, broader incident impact | | use-case-owned serving | independent change and isolation | duplicate cost, definitions and operational work | | full history retention | reproducibility and investigation | privacy, security, storage and deletion burden | | shorter retention | lower exposure and cost | weaker replay, evaluation and audit evidence | | human review | lower action authority and contextual judgment | delay, reviewer variation, fatigue and capacity limits | | automated action | speed and scale | stronger evaluation, policy, observation and recovery needs |

Make each trade-off testable. If centralization is chosen for consistency, measure publication lead time and domain-owner participation. If real-time processing is chosen for value, measure whether decision outcomes actually degrade with batch latency. If a human approval step is required, measure reviewer agreement, delay, escalation, and whether the reviewer sees enough evidence to make a meaningful decision.

Avoid false choices. A platform can centralize identity, metadata, templates, policy, and observability while domains own semantics and publication. A system can use batch for most features and stream only the signals that materially change the decision. A shared retrieval service can isolate indexes and policy by tenant or sensitivity.

Record the selected option, alternatives considered, evidence, assumptions, expected benefit, accepted cost, risk owner, and review trigger. Revisit the decision when traffic, data, model, regulation, team structure, provider capability, or consequence changes.

18. Establish the operating model

Data readiness is an ownership problem as much as a platform problem.

The platform team owns paved roads, runtime, templates, common security, observability, reliability, and recovery. Domain producers own source meaning, contract, correction, and quality. AI product teams own the decision, evaluation, model or prompt, policy, release, and outcome. Governance defines classification, permissible use, retention, access, risk acceptance, and exception paths. Consumers own appropriate use and downstream effects.

Give every production dataset, feature, index, model, evaluation set, and policy a named owner and support expectation. Shared ownership without named decisions becomes no ownership during an incident.

Use product-level service objectives for freshness, availability, correctness, and recovery. Combine them with decision-level quality and risk indicators. An available endpoint serving stale or unauthorized data is not a healthy AI product.

19. Migrate by one decision path

Do not pause all AI work until a universal data platform is complete. Choose one valuable decision with manageable consequence and build the trust path end to end.

Baseline the current decision and data. Preserve source evidence. Define contracts and authority. Build a representative evaluation set. Implement one production-like serving path with identity, policy, observability, cost, and recovery. Run in offline, shadow, or advisory mode before increasing authority.

Reconcile outputs against current practice and later outcomes. Explain differences. A new system is not automatically correct because its model score is higher. It may use different populations, time windows, labels, exclusions, or action thresholds.

Retire temporary copies, pipelines, features, indexes, and credentials after the path stabilizes. A pilot that leaves unowned data and infrastructure behind has created platform debt rather than readiness.

20. Use a production readiness gate

The release gate should answer whether the specific decision can operate safely under its intended authority, not whether the team completed a model-development checklist.

Review decision scope, data authority, contracts, lineage, point-in-time correctness, quality slices, privacy, security, evaluation, robustness, serving, freshness, feedback, drift, cost, fallback, incident response, recovery, documentation, and owner readiness.

Define hard stops. Examples include unknown data ownership, unresolved tenant leakage, evaluation leakage, missing high-consequence cases, inability to reproduce the dataset, unbounded external action, no safe fallback, absent incident authority, or no way to remove restricted data.

Define a bounded evidence window after release. Some behavior appears only after seasonality, delayed labels, content changes, index refresh, provider updates, or a full business cycle. Expansion should depend on observed results rather than a calendar date.

Production readiness does not mean the AI system is permanently approved. It means a named version is approved for a bounded decision, authority, population, environment, and evidence window.

21. Build a remediation roadmap from evidence

Classify gaps by decision consequence and dependency. A missing enterprise catalogue may not block one bounded path if its sources and owners are explicit. Missing point-in-time history may fundamentally block a historical evaluation.

Separate four types of work:

1. Decision changes reduce authority, narrow population, add review, or remove unsupported outcomes.
2. Data-product changes establish source ownership, contracts, quality, lineage, history, privacy, and access.
3. Platform changes provide repeatable ingestion, transformation, metadata, serving, observation, and recovery.
4. Operating changes establish owners, reviews, incident response, feedback, cost control, and lifecycle governance.

Sequence the smallest set that makes one decision trustworthy. Avoid launching a multi-year platform transformation without a measurable consumer path. Also avoid accumulating one-off pipelines that cannot share identity, lineage, quality, or operating controls.

Track each gap with evidence needed, owner, due date, consequence, workaround, expiry, and release implication. Close the gap only when the evidence exists in the intended operating environment.

22. Readiness checklist

items={[ 'The AI decision, user, subject, authority, consequence, horizon, ground truth, recovery, and acceptance evidence are explicit.', 'Every material data element has an authoritative source, owner, meaning, time semantics, classification, and known limitation.', 'Versioned data contracts cover schema, semantics, keys, quality, freshness, change, publication, and incident ownership.', 'Source evidence, transformation revision, parameters, checkpoints, results, and publisher make important datasets reproducible.', 'Quality thresholds are tied to the decision and measured across important slices and time.', 'Historical evaluation uses point-in-time-correct data and controls identity, temporal, label, and retrieval leakage.', 'Purpose, privacy, minimization, retention, access, deletion, derived copies, and external AI tools are governed.', 'Training, evaluation, retrieval, serving, feedback, and audit datasets are separated and versioned for their purposes.', 'The evaluation set includes representative, difficult, rare, harmful, refusal, and recovery cases without benchmark overfitting.', 'Features and retrieval preserve version, freshness, access policy, source authority, and offline-to-online consistency.', 'End-to-end freshness, latency, failure, fallback, and partial-source behavior have objectives and tests.', 'Feedback distinguishes interaction, expert review, operational outcome, correction, and incident, including delayed truth.', 'Monitoring covers data, concept, quality, policy, cost, outcome, and important slices with owned response thresholds.', 'Platform, producer, AI product, governance, consumer, incident, and risk-acceptance responsibilities are named.', 'The release is bounded by version, population, authority, environment, evidence window, stop conditions, and recovery.', ]} />

Limitations and tailoring

This standard cannot determine whether an AI use case is lawful, ethical, safe, accurate, or valuable without its complete context. It is not legal, regulatory, privacy, security, statistical, or domain-specific advice. High-consequence decisions require qualified reviewers and may require controls beyond those described here.

Perfect historical data may be impossible. Ground truth can be disputed. Representative evaluation can conflict with privacy minimization. Full reproducibility can be limited by external providers, non-deterministic systems, or unavailable source history. Document these limits, reduce authority, add review, improve observation, or decline the use case when uncertainty exceeds the organization’s risk tolerance.

A modern lakehouse, warehouse, feature store, catalogue, vector database, or orchestration platform can improve implementation. None guarantees readiness. Conversely, a modest platform can safely support a bounded AI decision when ownership, contracts, evidence, evaluation, policy, and operations are strong.

The readiness question should remain concrete: can this named decision operate for this population, under this authority, from these data versions, with this evidence and recovery model? If the organization can answer and prove that question, it has a foundation for production AI. If it cannot, the missing evidence defines the next useful investment.

Reassess that answer after material source, policy, model, provider, population, workflow, or authority changes. Readiness is maintained through evidence and operation. It is not a one-time platform certification.