AI Governance Framework: Production Decisions, Controls and Evidence

Build proportionate AI governance with a decision inventory, a worked routing example, release evidence, calibrated monitoring and explicit stop conditions.

Govern the decision, not just the model

A production AI governance framework should identify what the system may decide, who is accountable, what evidence permits release, and how someone can stop or challenge a harmful outcome. The control level should follow the consequence and reversibility of the decision, not whether the model is called an agent, classifier or copilot.

Start with one use case and a named owner. A model inventory without decision authority is incomplete: the same model can draft an internal note or trigger a customer-facing action, with different risks.

This guide is for engineering and product teams designing those controls. The examples are hypothetical, not Ampity customer results. It addresses operational governance, not legal advice, certification or a determination that a regulated use is lawful.

The NIST AI Risk Management Framework is a voluntary framework for managing AI risks. It is a useful organizing reference, not a substitute for the obligations that apply to a particular organization, jurisdiction or use case.

Write a decision inventory that changes the design

An inventory entry should describe the full system: model, data, retrieval sources, user interface, integrations and people. Record both intended use and foreseeable misuse. “Human in the loop” is not a sufficient control description unless that person has time, evidence and authority to disagree.

| Inventory field | Question to answer | Design consequence | |---|---|---| | Decision and affected party | What changes for whom after this output? | Defines the outcome to test, including harm outside the model metric | | Authority | Can the system suggest, route, publish or execute? | Determines which actions need service controls and approval | | Inputs and access | Which data is necessary, permitted and available to this user? | Limits collection, retrieval and retention | | Failure and reversibility | What happens if the answer is wrong or unavailable? | Determines fallback, escalation and recovery | | Evidence and uncertainty | What data supports the intended use, and where is coverage weak? | Defines release tests and unsupported cases | | Ownership | Who accepts the residual risk and who can stop the system? | Makes release and incident decisions actionable | | Change triggers | What new data, model, purpose or permission requires review? | Prevents an approved pilot from silently gaining new authority |

Do not reduce all of this to a single risk score. A rare irreversible action can need stronger controls than a frequent, easily corrected suggestion. Where the decision affects legal rights, safety or access to essential services, involve the relevant domain and legal specialists before selecting the controls.

For a bounded internal assistant, the service owner may approve ordinary updates within an agreed policy. A new external action, sensitive data source or affected population should reopen the review. A committee can resolve cross-functional decisions, but it should not obscure who owns the operational response.

Worked example: govern a support-ticket routing model

Suppose a software company wants a classifier to route incoming tickets to its billing, technical or account-access queue. The business objective is correct assignment without delaying urgent access problems.

The first release proposes a queue to an operator. It cannot close tickets, change account permissions or send a customer message. The routing service preserves the original ticket and the operator can correct the suggestion.

The release decision needs more than overall classification accuracy:

| Observed issue in the hypothetical pilot | Governance decision | Evidence required before expansion | |---|---|---| | Account-access tickets are sometimes sent to billing | Retain a deterministic route for recognized lockout signals and an operator escalation path | Test missed urgent cases and time to reassignment, not just average accuracy | | One supported language has few labeled examples | Keep that language in manual routing until its intended use is evaluated | A reviewed sample, label-quality check and documented coverage limits | | Operators accept suggestions without opening the ticket | Change the review interface and inspect actual override behavior | Observation that reviewers can find the evidence and correct a bad route | | Tickets contain credentials or sensitive attachments | Exclude unnecessary fields before model processing | Tests of minimization, access restrictions and logging behavior | | A team proposes automatic ticket closure | Treat closure as a new decision, not a routing-model update | A separate impact review, authority boundary and recovery design |

These are design choices for this example, not mandatory controls for every classifier. The routing rule can also be wrong, and manual handling has its own error rate. Compare the proposed system with the existing process using the same cases and outcome definitions.

A release record might therefore say: “Queue suggestions only, evaluated languages only, no closure permission, operator correction retained, service owner responsible for routing incidents.” That is more testable than “responsible AI approved.”

Test outcomes across relevant groups

Choose measures that correspond to the decision. For routing, distinguish the cost of a missed urgent ticket from a routine reassignment. Inspect relevant slices such as supported language, ticket source and issue type instead of relying only on an aggregate result.

Define each numerator, denominator, label source and observation period. A small slice with very few known outcomes cannot support the same conclusion as a large, well-labeled sample. Record uncertainty and label disagreement rather than treating a single measured gap as proof of either fairness or discrimination.

Where demographic analysis is appropriate, determine the lawful and necessary data collection with qualified specialists. Do not collect sensitive attributes simply because a generic checklist asks for them. Omitting an attribute also does not prove that other inputs lack proxy effects.

If a mitigation changes thresholds, training samples or outputs, rerun the relevant outcome and slice tests. Improvements in one measure can worsen another. Record the tradeoff, the alternatives considered and who accepted the remaining limitation.

Make explanations useful for the person receiving them

An explanation should answer a real question: why an operator saw a recommendation, which evidence informed it, or how someone can request correction. A technical feature-attribution chart and a user-facing explanation serve different purposes.

NIST's Four Principles of Explainable Artificial Intelligence separates providing an explanation from whether it is meaningful, accurately reflects the system, and communicates its knowledge limits. Apply those distinctions when reviewing an explanation interface.

For the routing example, show the suggested queue, relevant ticket evidence, known coverage limits and a correction control. Do not generate a confident narrative that claims to reveal the model's exact reasoning when that claim has not been established.

Feature-attribution methods such as SHAP or LIME may assist investigation. Selecting one does not establish regulatory compliance, prove causation or ensure the explanation is suitable for an affected person. The SHAP maintainers' causal-interpretation example shows why a predictive relationship should not be read as the effect of an intervention. Validate the explanation for its intended audience and obtain domain-specific review where required.

Calibrate monitoring to a response, not an arbitrary threshold

Input drift, changed predictions and worse outcomes are different signals. A shift in ticket topics during a product launch may be expected. A stable input distribution can still contain harmful errors.

| Signal | What to compare | Response to define in advance | |---|---|---| | Input or retrieval change | Current data against a relevant reference period, accounting for seasonality and collection changes | Investigate the changed source and coverage; restrict unsupported inputs if necessary | | Outcome deterioration | Confirmed results and error costs, including important slices | Route affected cases to fallback while the owner investigates | | Control failure | Unauthorized action, unavailable approval or missing access check | Disable the affected capability and preserve incident evidence | | Review failure | Unhandled escalations, stale queues or ineffective overrides | Reduce automation or restore manual capacity | | Dependency change | Model, prompt, feature pipeline, policy or source version | Run the change-specific evaluation before expanding traffic |

There is no universal performance-drop or fairness-gap threshold that makes every system unsafe. Set an alert around a decision the operator can make, using the baseline, consequence of error, volume, uncertainty and available response capacity.

If using a statistical test, name the null hypothesis, test assumptions, sample window and significance level. A p-value above a chosen significance level is not the usual evidence for rejecting that null; see NIST's explanation of critical values and p-values. Also consider effect size and repeated testing across features or time windows. Statistical significance alone does not show operational harm, and failure to detect a change does not prove safety.

Outcome labels may arrive late. Use early signals to investigate, but keep checking actual outcomes when labels become available. Do not automatically retrain and promote a model just because a drift alert fired.

Retain enough evidence to investigate, but not every input

Link an outcome to the deployed model or provider version where available, application configuration, relevant data or source version, authorization decision, human approval and final disposition. For a provider that does not expose an immutable model version, document that reproducibility limitation.

Decide which sensitive inputs need retention, who can access them, and when they expire. Hashing an identifier is not automatically anonymization; a predictable value may still be linkable or guessable. Minimize logs and protect the retained evidence according to its sensitivity.

A model rollback does not reverse actions already taken. The response plan must also cover reassignment, correction, notification or another appropriate remedy for affected cases. Test the fallback under a realistic load before relying on it.

Use this release and change-review checklist

Before release, the accountable owner should be able to point to:

• A permitted-use statement, excluded uses and application-enforced action boundaries.
• A baseline comparison, reviewed evaluation cases and documented coverage gaps.
• Data-access and retention decisions, including third-party processing where relevant.
• A working correction or escalation route with adequate human capacity.
• Monitoring definitions, response owners and capability-specific stop conditions.
• A tested fallback, incident record and process for addressing past outcomes.
• Named approvals and the changes that invalidate them.

The next useful step is to complete that record for one production decision, then trace each claimed control to a test or operational artifact. An empty evidence field is a work item, not something to hide behind a framework label.

For help defining evaluation evidence and production monitoring, see Ampity's LLM evaluation and observability service. The data-readiness evidence checklist covers the source, access, retrieval and deletion controls that should feed that governance record. Neither replaces the domain or legal review a specific decision may require.