Production-Grade AI Systems

Release and operate AI with versioned data and model contracts, task-specific evaluation, bounded authority, monitored outcomes, and rehearsed fallback.

trigger="A predictive or generative model is ready for evaluation in a specific production workflow." owner="The engineering owner accountable for the workflow's user and business outcomes." participants={["Product and process owner", "AI or ML engineer", "Data owner", "Domain reviewer", "Security and privacy reviewers", "Operations lead"]} prerequisites={[ "A defined task, permitted actions, current-process baseline, and non-model fallback.", "Approved data and model rights, representative evaluation cases, and known failure costs.", "Versioned application and model dependencies, controlled release access, and operational ownership." ]} outputs={[ "A release manifest and evaluation report with subgroup coverage and known limitations.", "A monitored pilot with an authority boundary, cost limits, and human escalation.", "A tested disable, rollback or forward-recovery, and downstream reconciliation runbook." ]} doneWhen={[ "The domain owner accepts task-specific quality and consequential failure behavior.", "Security and data owners accept permissions, data handling, retention, and provider boundaries.", "The system meets its service objectives under load and dependency failure.", "Operators can stop model use and reconcile affected work without silently repeating external actions." ]} />

A production result is more than a successful prediction

Consider a support classifier that produces valid labels but routes urgent customer issues into the wrong queue. Its endpoint is healthy while the workflow is failing. A generative assistant can similarly return fluent text that lacks evidence or exceeds its authority.

Start with one decision or task and its consequences. This playbook covers release and operation after the use case is defined. It does not claim that a particular fraction of AI projects fails, that every model needs the same infrastructure, or that an offline score certifies production readiness.

Use a simple baseline: the existing process, a deterministic rule, or the current deployed model. An AI system should earn its additional cost and failure modes with evidence relevant to the task.

1. Define the task, authority, and acceptance contract

The process owner records inputs, expected outputs, permitted actions, users, affected people, and prohibited uses. Separate advice, drafting, classification, and execution. A model's confidence or fluent explanation does not establish permission to read data or perform an action.

| Gate | Accountable participant | Evidence | | --- | --- | --- | | Task value | Product or process owner | Comparison with the current workflow, including rework | | Output quality | Domain reviewer | Rubric, error costs, and evaluated examples | | Data use | Data and privacy owners | Rights, provenance, retention, and provider terms | | Access and actions | Security owner | Permission and denied-action tests | | Reliability | Service owner | Deadline, load, quota, and dependency-failure tests | | Operational release | Operations lead | Alerts, fallback, escalation, and recovery exercise |

Choose acceptance limits from harm, expected benefit, and operating constraints. A universal accuracy target, latency figure, or confidence cutoff cannot serve every use case.

The NIST AI Risk Management Framework offers a voluntary structure for governing, mapping, measuring, and managing AI risk. Use it to organize responsibility and evidence, not as a certification or substitute for applicable legal review.

Gate: the team can identify who is authorized to approve the workflow and who can stop it. Unresolved authority is a release blocker.

2. Version the complete prediction or generation path

For a predictive model, bind the artifact to training code, data references, feature definitions, preprocessing, runtime dependencies, thresholds, and evaluation results. For a hosted generative system, include the available provider/model identifier, prompts, retrieval configuration, tool schemas, safety policy, and relevant application revision.

Do not claim exact reproducibility when a provider does not expose or guarantee every underlying version. Record that limitation and use regression evaluation and change monitoring appropriate to the dependency.

"type": "svg-architecture", "title": "A release is a versioned workflow, not one model file", "nodes": [ ], "links": [ ], "caption": "Model output remains untrusted. The application enforces permissions and action authority independently. Evidence collection also needs data minimization, access control, and retention." }} />

The registry or release catalog needs immutable identifiers and approval records, not indefinite storage of every artifact. The data and operations owners set retention from recovery, audit, privacy, licensing, and storage requirements. Large models and training-derived artifacts may be costly or sensitive.

Retain enough approved artifacts and metadata to recover supported releases. Verify that required keys, runtimes, feature definitions, and provider access remain available. A model file alone is not a usable rollback.

3. Test input quality and training-serving consistency

The data owner defines schema, ranges, missing-value behavior, timestamps, freshness, and allowed sources. Test late, duplicated, malformed, unauthorized, and unavailable inputs. Decide which cases are rejected, deferred, or handled by an approved fallback.

For predictive models, use point-in-time data appropriate to the prediction. Prevent future outcomes or post-decision fields from leaking into training. Compare preprocessing, defaults, time zones, rounding, and feature semantics between development and serving.

Scikit-learn's common-pitfalls guidance addresses inconsistent preprocessing and data leakage. Pipelines can help apply learned transformations consistently, but evaluation design and data availability still need review.

A feature store or shared library is an option, not a guarantee. Validate actual values on representative records across paths and confirm freshness. Shared code can still receive different source data or versions.

For generative systems, test retrieval permissions, source freshness, missing evidence, and hostile instructions in retrieved material. A retrieved document is input data, not authority to override system policy. Domain reviewers must assess whether the output is supported, not only whether it sounds plausible.

4. Build an evaluation that reflects the decision

The AI owner separates development cases from held-out release evaluation. Include routine work, rare important failures, unanswerable cases, boundary conditions, and foreseeable misuse. Version the datasets and document their source and coverage.

Measure task outcomes and error categories. A classifier may need precision and recall for a costly category; a drafting tool may need factual support, instruction adherence, and reviewer correction effort. Do not collapse materially different failures into one overall score.

Evaluate relevant subgroups when justified and lawful. Record sample size, uncertainty, known gaps, and the consequences of uneven performance. Small samples cannot support confident claims that every group is adequately represented.

Use a qualified human review process for subjective or consequential outcomes. If model-based grading is used, validate it against reviewed examples and monitor disagreement. An automated grader's score is evidence to assess, not final authority.

Gate: release requires acceptance of both measured results and residual limitations. A better average does not cancel an unacceptable failure on a critical workflow.

5. Prove the serving and fallback contract

The service owner measures end-to-end response or completion time, queueing, throughput, errors, resource use, dependency limits, and cost per useful outcome. Include retries, review work, rejected outputs, and fallback cost.

Choose embedded, dedicated-service, managed, batch, or asynchronous serving from measured requirements. Avoid vendor latency rankings detached from model size, hardware, batching, input length, and network conditions.

Treat quantization, batching, caching, distillation, and runtime export as separate experiments. Compare quality, subgroup behavior, memory, throughput, and latency against the approved baseline. None has a universal speedup or accuracy-loss bound.

The fallback must preserve the task's safety contract. It may be a deterministic rule, human review, deferred processing, or an explicit unavailable response. A model-generated confidence value is not enough to decide that a consequential action is safe.

Test provider outage, quota exhaustion, slow response, partial output, malformed output, and failure after an external action. Use stable operation identifiers and reconcile ambiguous results before retrying any consequential effect.

6. Run a bounded release with meaningful review

The release owner first validates the whole path in an approved environment. Shadow execution can compare behavior without displaying new outputs, but it still uses data and resources. Disable actual external writes in the shadow path and obtain provider and privacy approval.

For a live pilot, select the cohort and allocation unit to avoid inappropriate interference or inconsistent treatment. Define the primary outcome, guardrails, sample plan, observation period, and stopping rules before examining results. There is no universal traffic percentage or two-week experiment duration.

Monitor quality, human review load, service behavior, and cost together. Keep reviewer capacity realistic. A requirement for human approval is ineffective if reviewers lack the context, time, or authority to reject the result.

"type": "flow", "title": "Evidence gates for an AI release", "steps": [ ], "caption": "A passing model evaluation is one gate. Production use also requires operational, data, security, and domain acceptance." }} />

7. Separate model evaluation from system evaluation

A model comparison answers only part of the release question. System evaluation must include the instructions, retrieval or features, policy checks, output validation, user interface, human review and downstream effects that shape the real workflow.

Create an evaluation manifest with the dataset version, task rubric, grader configuration, model and parameters, prompt or feature pipeline, tool contracts, application revision and test environment. The OpenAI Evals API is one example of representing testing criteria and a data-source schema as a reusable evaluation. The provider-specific implementation may vary, but the evidence must remain comparable and versioned.

Separate blocking checks from scored quality. Permission bypass, an unauthorized external action, invented evidence or a duplicate transaction should not pass because the remaining examples score highly. Report those failures as release blockers with reproduction evidence and an accountable owner.

For human review, define the decision labels, examples and disagreement path. Measure whether reviewers agree on the consequential cases. If reviewers cannot identify a correct result from the available information, improve the task definition or source evidence before using the model score to justify release.

8. Threat-model the application boundary

The security review traces untrusted inputs through the complete system. For generative applications, include direct and indirect prompt injection, insecure output handling, sensitive information disclosure, excessive tool authority and poisoned retrieval content. The OWASP Top 10 for LLM and Generative AI applications provides a current risk taxonomy, not a certification checklist.

Keep secrets, identity, permissions, spending limits and action approval outside prompts. Validate structured output before use. Authorize every tool call in application code against the authenticated user, workflow state and operation policy. Give the model the narrowest data and action scope required for the task.

Exercise an attack that asks the model to reveal another user’s information, alter its policy, call a disallowed tool and repeat an ambiguous action. Confirm that the system blocks the effect even if the model follows the hostile instruction. A refusal from the model is helpful, but the application control is the security boundary.

9. Establish cost and capacity controls

Budget the complete workflow: model and embedding use, retrieval, storage, evaluation, observability, human review, retries, fallback and incident handling. Track cost per useful completed task and per reviewed exception. A cheaper model call can produce a more expensive workflow when correction and escalation increase.

Set explicit limits for input size, retrieved context, model output, tool calls, retries and concurrent work. Treat quota exhaustion and cost spikes as operating conditions with alerts and a safe degradation path. Caches must preserve identity, data retention and version boundaries.

Capacity testing should include the longest supported input, peak concurrent requests, slow dependencies, evaluator load and the backlog created during an outage. Prove that recovery does not overwhelm the provider or repeat external effects. Scale decisions should follow these observed constraints rather than a generic token or request estimate.

10. Investigate drift as a hypothesis, not a diagnosis

Track input quality, feature distributions, prediction distributions, performance against reliable labels, service health, and business outcomes as distinct signals. Google's model-monitoring documentation distinguishes distribution-based monitoring from other evaluation needs. Confirm the capabilities of the deployed monitoring version.

A conversion or revenue decline does not prove model drift. Seasonality, traffic mix, product changes, an experiment, broken logging, delayed labels, pricing, or an operational failure can produce the same observation.

The AI owner follows a diagnosis sequence:

1. Verify signal integrity, label delay, sample composition, and recent releases.
2. Check input pipeline health, feature freshness, schema changes, and serving errors.
3. Compare cohorts and time periods that can support a meaningful conclusion.
4. Examine distribution changes alongside task-quality evidence.
5. Select a repair, rollback, retraining experiment, or no-change decision with an owner.

Calibrate statistical alerts to the feature, sample size, multiple comparisons, seasonality, and cost of investigation. Do not copy universal PSI or p-value cutoffs into every service. A drift alert triggers investigation; it should not automatically promote a retrained model.

11. Minimize sensitive evidence and rehearse incidents

Logging every input and output can create a new sensitive-data repository. The data owner defines what is necessary for evaluation and investigation, whether redaction or controlled sampling is appropriate, who may access records, and how deletion and retention work.

Preserve version IDs, outcome labels, error categories, timing, and operation references where useful without assuming raw payload retention is required. Test whether redaction removes data needed for a critical investigation and resolve that tradeoff explicitly.

The incident owner defines escalation for harmful output, data exposure, unauthorized action, systematic subgroup failure, and repeated unsupported answers. Operators need a tested disable control and a safe work queue for cases that cannot continue automatically.

On rollback, restore a compatible combination of model, preprocessing, retrieval, prompts, thresholds, and application contracts. If the previous release is no longer safe or available, use the approved non-model or forward-recovery path. Reconcile affected records and external actions; changing a model does not undo completed decisions.

Recovery exercises to complete before release

The incident owner assigns an operator and an observable pass condition to each relevant exercise. Use protected test cases and disable real external effects unless the exercise has explicit production authorization.

| Failure to inject | Expected containment | Evidence required to resume | | --- | --- | --- | | Provider quota or endpoint outage | Bound retries and switch to the approved fallback | Deadline, backlog, and duplicate-action checks | | Missing or stale input feature | Reject, defer, or use the documented missing-data path | Source freshness and training-serving consistency | | Unsupported answer or harmful output | Restrict the workflow and route affected cases for review | Domain-reviewed regression cases and scope assessment | | Permission loss during a request | Prevent unauthorized retrieval, delivery, or action | Current-policy checks and negative access tests | | Model action times out after acceptance | Stop blind retries and inspect the operation record | Downstream reconciliation using stable operation identity | | Prior model or provider version unavailable | Use the approved non-model or forward-recovery path | Compatible inputs, outputs, and operator acceptance |

Set recovery objectives from the workflow's consequences and support capacity. An operator must know when to pause processing, how to preserve pending work, and who can approve resumption. A successful model endpoint health check does not prove that the queue, reviewers, or downstream system can safely resume.

12. Use a release record and acceptance checklist

Workflow, permitted actions, and accountable owner:
Baseline, intended benefit, and critical failure costs:
Data rights, source versions, and provider constraints:
Model and complete processing-path identifiers:
Evaluation set, rubric, subgroup coverage, and uncertainty:
Performance, operating cost, and fallback evidence:
Pilot allocation, stopping rules, and reviewer capacity:
Monitoring signals and drift-investigation procedure:
Retention, access, deletion, and incident controls:
Rollback compatibility and downstream reconciliation:
Approvers, unresolved limitations, and review trigger:

"The task and action authority are explicit, with a non-model operating path.", "Data rights, provider terms, permissions, and retention have accountable approval.", "The complete processing path is versioned and recovery dependencies are available.", "Evaluation covers consequential failures, relevant groups, and uncertainty.", "Serving tests cover load, quotas, malformed results, and dependency outages.", "Pilot reviewers have real capacity to reject or correct outputs.", "Business decline is investigated without automatically labeling it model drift.", "Rollback or disable procedures reconcile completed actions and pending work." ]} />

Limitations and approval boundary

Held-out evaluation cannot cover every future input or adversarial condition. Delayed labels, small subgroups, model-provider changes, and limited reviewer capacity can leave material uncertainty. Record the remaining gaps and choose a narrower scope or stronger fallback when the evidence cannot support the intended use.

This guide covers general predictive and generative workflows. Regulated or consequential decisions can require additional legal, domain, accessibility, and human-oversight controls. The accountable owners must determine their applicability; neither a framework reference nor an engineering checklist establishes compliance.

The output is a release evidence pack and an operating contract. Editorial review does not constitute domain approval, factual certification, or authorization to deploy the system.