Production AI Evaluation Workbook

A practical workbook for defining AI task contracts, building representative evaluation sets, writing rubrics, adjudicating results, setting release gates, and...

trigger="An AI feature is moving beyond demonstration, a model or prompt change may affect production behavior, or the team cannot explain what evidence permits an AI output to be displayed, recommended, approved, or executed." owner="One product owner accountable for the task outcome, paired with one evaluation owner responsible for the dataset, rubric, run integrity, and result record. Domain, security, privacy, and operations owners retain decisions in their remit." timebox="Two weeks for the first bounded task and release gate. Maintain the workbook for every material model, prompt, retrieval, tool, policy, data, or threshold change." participants={["Product owner", "Evaluation owner", "Domain reviewer", "AI or application engineering", "Security", "Privacy or data owner", "Operations", "Affected workflow owner", "Independent reviewer for consequential cases"]} prerequisites={[ "One bounded task and intended user, with the current manual or deterministic baseline described.", "A named consequence if the system is wrong, delayed, unavailable, manipulated, or over-trusted.", "Approved access to representative examples and a retention policy for evaluation inputs, outputs, traces, and reviewer decisions." ]} outputs={[ "A task and risk contract with allowed autonomy, abstention, escalation, and stop conditions.", "A versioned evaluation set with slice coverage, provenance, permissions, and held-out cases.", "A scoring rubric with observable criteria and an adjudication guide.", "A candidate report covering quality, safety, security, latency, cost, tool behavior, and reviewer disagreement.", "A bounded release plan and production-feedback process tied to the next evaluation version." ]} doneWhen={[ "The evaluation measures the complete AI system against the real task rather than grading model fluency alone.", "Critical slices and unacceptable outcomes have explicit thresholds even when the overall average passes.", "Reviewers can reproduce disputed decisions from the case, evidence, rubric, and system version.", "The release limits permissions and exposure according to risk and has tested stop, fallback, and rollback paths.", "Production corrections, incidents, abstentions, and drift signals feed a controlled evaluation update." ]} />

Define the decision before the metric

Start with the production decision the evaluation must support. Examples include whether a support answer may be shown, a forecast may enter planning, a document extraction may update a record, or an agent may execute a bounded tool call. “Evaluate the model” is not a release decision.

Write the user, task, starting information, expected output, allowed evidence, prohibited behavior, maximum delay, fallback, and accountable human. State what the system may do without review and what always requires approval. An answer assistant, ranking system, recommendation, and action-taking agent need different evidence.

Describe the current baseline. It may be a manual process, rules engine, search interface, older model, or no capability. Measure completion, correction effort, elapsed time, safety failures, cost, and abandonment where the evidence is available. The candidate should improve a real outcome, not only a benchmark score.

NIST’s AI Risk Management Framework treats evaluation as part of design, deployment, use, and ongoing risk management. Use that lifecycle perspective here. The workbook is not a one-time test report.

Record the task and risk contract

Complete one row for every materially different task:

| Field | Workbook entry | | --- | --- | | task | one observable job the system supports | | affected actor | user, operator, customer, supplier, or public | | business consequence | what changes if the output is accepted | | evidence boundary | sources the system may use | | action boundary | tools and resources it may reach | | unacceptable outcome | harm or loss that blocks release | | abstention | when the system must decline or defer | | escalation | reviewer and required context | | expiry | when the output is too old to act on | | recovery | how an incorrect output or action is contained and corrected |

Classify consequence separately from frequency. A rare recommendation that can stop a supply route, approve money, expose private data, or change access deserves stricter evaluation than a frequent draft with easy review.

Define the human role precisely. “Human in the loop” can mean reviewer, approver, operator, appeal owner, monitor, or incident responder. A person who lacks time, authority, evidence, or a usable interface is not an effective control.

Build a representative case inventory

List the dimensions that change task difficulty or risk. These can include user role, organization, language, source quality, document format, ambiguity, time sensitivity, amount, region, product category, new versus known entity, missing evidence, conflicting evidence, and adversarial input.

Create slices before sampling cases. A favorable average can conceal failure for a small but consequential group. Name critical slices whose threshold cannot be traded against easy cases.

Include:

• routine cases that reflect normal volume;
• boundary cases near policy, amount, time, or confidence limits;
• ambiguous cases with more than one plausible interpretation;
• missing and conflicting evidence;
• requests that must be denied or escalated;
• outdated, malformed, duplicated, and multilingual input where relevant;
• prompt injection, poisoned retrieved content, and malicious tool output;
• dependency timeout, empty retrieval, unavailable tool, and partial action;
• previously corrected production cases after privacy review.

Do not copy raw production conversations into an evaluation file by default. Record provenance, purpose, authorization, minimization, access, retention, and deletion. Use synthetic cases for hazards that should not be recreated with real sensitive data.

Version the evaluation set

Give the set an immutable version and record case identifier, task, slice labels, input reference, expected evidence, expected outcome, risk class, source, reviewer status, and last review date. Separate development cases from a held-out release set.

Prevent candidate authors from tuning repeatedly against every release case. Keep a small hidden or independently administered set for detecting overfitting where the process warrants it. Rotate cases carefully so trend comparisons remain interpretable.

Track coverage by decision-relevant slice, not by raw case count. Fifty near-identical easy examples do not compensate for a missing denial or high-consequence case. Explain gaps and assign owners.

When production evidence adds a case, preserve the original event reference and a sanitized evaluation representation. Record whether the failure came from data, retrieval, reasoning, tool selection, policy, user interface, reviewer behavior, or downstream execution.

Write the rubric before the candidate run

The rubric should describe observable evidence. “Good answer” is not reproducible. For a recommendation, criteria might include correct subject, current evidence, calculation, uncertainty, policy compliance, actionability, and safe escalation. For an agent, add tool choice, arguments, authorization, sequence, duplicate protection, and final state.

Use pass, fail, abstain, and escalate where those states reflect product behavior. Do not force every uncertain case into correct or incorrect. Define whether a safe abstention is accepted and how excessive abstention affects usefulness.

Weighting cannot erase a critical failure. Use hard gates for data exposure, unauthorized action, fabricated approval, unsafe advice, duplicate transaction, or another unacceptable outcome. Then use weighted or aggregated scores for lower-consequence quality dimensions.

Provide positive, negative, and borderline examples. Ask two qualified reviewers to score a calibration subset independently. Discuss disagreement and revise the rubric, not the score, when wording caused inconsistent interpretation.

Capture the complete system version

An AI result depends on more than the model name. Record model and provider identifier, prompt or instruction version, retrieval corpus snapshot, chunking and ranking configuration, tool definitions, policy rules, output validators, memory behavior, temperature and other relevant parameters, dependency versions, and feature flags.

Pin what can be pinned. Where a provider can change behavior behind an identifier, record the date, returned model identifier, request configuration, and observed evidence. Treat a material provider update as a trigger for targeted re-evaluation.

Evaluate the production path. A notebook response without retrieval permissions, tool authorization, parser, timeout, cache, guardrails, or user interface is not the system users will receive.

Retain trace data proportionately. Keep enough to explain source selection, output, tool proposal, policy decision, reviewer action, latency, token or compute use, and final state without collecting unnecessary sensitive material.

Run deterministic tests first

Use ordinary tests for behavior that should be deterministic: schema validation, authorization, tenant isolation, allowed tool list, argument constraints, idempotency, budget admission, timeout, state transitions, audit fields, redaction, and fallback routing.

Inject unavailable dependencies and malformed responses. Confirm that a failed validator stops the protected action. Test that approval binds to the exact action, resource, actor, version, and expiry, and that a changed proposal requires new approval.

The OWASP Generative AI Security Project describes excessive agency as excessive functionality, permissions, or autonomy. Its Excessive Agency guidance supports testing the reachable actions and permissions, not merely asking whether the model intends to behave.

Fail the run if a deterministic safety control is unreliable. A model-quality score cannot compensate for an authorization bypass or duplicate external effect.

Run and preserve candidate evidence

Run every candidate against the same eligible case-set version, rubric, dependency fixtures, and resource limits. Record failures and timeouts rather than silently rerunning only unsuccessful cases. If a service outage invalidates a run, label it invalid and repeat the whole affected comparison.

For each case retain system version, output, citations or evidence, tool proposals and receipts, policy result, latency, complete cost where available, automatic scores, reviewer scores, disagreement, and final disposition.

Use model-based graders for scale only after validating them against qualified human judgement on the task. A grader can share biases, miss context, or prefer the style of one candidate. Keep deterministic checks and high-consequence human review independent.

The OpenAI Evals API is one implementation option for versioned data sources, criteria, runs, and graders. The workbook’s contract remains provider-neutral so evidence survives a change in model or platform.

Adjudicate disagreement

Send critical failures, reviewer disagreement, low-confidence judgements, and sampled passes to adjudication. The adjudicator should see the task contract, case, permitted evidence, candidate output, trace, rubric, and reviewer reasons without being told which candidate the team prefers.

Record whether disagreement came from rubric ambiguity, missing domain information, evaluator error, product-policy conflict, or genuinely subjective judgement. Update the rubric only through a new version and re-score affected candidates when the change can alter the decision.

Measure reviewer agreement as a diagnostic, not a target to game. High agreement on a weak rubric is not proof of validity. Low agreement can reveal that the product has not defined acceptable behavior.

Protect reviewers from automation bias. Do not preselect the model’s preferred label as the default. Give them source evidence and a clear option to reject, abstain, or request more information.

Set slice-level release gates

Create a release table with overall usefulness, critical slice thresholds, hard safety gates, maximum abstention, latency, cost, reviewer effort, and recovery performance. State the minimum case count or confidence needed for each comparison.

| Gate | Example decision rule | | --- | --- | | unacceptable outcome | zero observed in release set, plus deterministic control test | | critical slice | meets its own threshold regardless of overall average | | usefulness | improves or meets baseline on accepted task outcomes | | abstention | stays inside the product’s review capacity and user promise | | latency | meets the task deadline at representative concurrency | | complete cost | includes retries, retrieval, tools, evaluation, review, and correction | | recovery | failed action or wrong output can be contained and reconciled |

Do not promote because one headline metric improved. Compare distributions and slices. Investigate whether a gain came from more abstention, longer answers, greater review effort, or a changed case mix.

Document the decision, dissent, exclusions, remaining risks, and expiry. A release gate is evidence for a bounded version and use case, not a claim that the model is generally safe or intelligent.

Measure repeatability, latency, cost, and review load

Run a representative subset more than once when the system is non-deterministic. Record how often the outcome class, cited evidence, tool choice, and action proposal change. A system can have a good average score and still be operationally difficult because the same input alternates between accept, abstain, and unsafe action.

Measure end-to-end latency at representative concurrency, including retrieval, reranking, model calls, tools, validators, retries, approval wait, and reconciliation. A fast model response does not help if a downstream tool queue misses the task deadline.

Calculate complete cost per accepted task. Include model and embedding use, retrieval infrastructure, tools, retries, observability, evaluation runs, human review, correction, and idle capacity. Compare cost by slice because long documents, uncommon languages, or repeated tool calls can create a different operating profile.

Estimate reviewer capacity from observed review time and arrival pattern. Define the maximum queue age and behavior when capacity is exhausted. The system may reduce intake, switch to read-only or suggestion mode, use a deterministic fallback, or tell the user that review is delayed. It must not silently bypass review.

Treat large variance as evidence. Investigate whether it comes from input ambiguity, unstable retrieval, model sampling, provider behavior, tool state, or an unclear rubric. Do not hide variance behind a single mean.

Design the bounded production release

Limit the first release by user group, organization, task class, action type, permission, amount, region, and time. Keep irreversible or high-consequence actions in proposal mode until their own evidence passes.

Define production stop signals: critical error, unsafe output, unauthorized tool attempt, correction rate, disagreement, abstention surge, latency, cost, dependency failure, complaint, or drift in input distribution. Name the person who can stop the feature without a committee meeting.

Provide a fallback that users can understand. This may be the previous system, manual review, read-only mode, no answer, or delayed processing. Test the transition and preserve unfinished work.

NIST’s Generative AI Profile highlights governance, pre-deployment testing, content provenance, and incident disclosure. Translate the relevant risks into release controls and evidence for this task.

Monitor outcomes, not model mood

Monitor the task outcome and control path: accepted completion, correction, override, escalation, abstention, policy denial, tool failure, duplicate prevention, latency, cost, and incident. Segment by the same critical slices used in evaluation.

Review sampled successes, not only complaints. Silent errors can look like user acceptance. Where downstream outcomes mature later, preserve cohorts and evaluate them only after the observation window.

Detect input and evidence changes. A stable model can fail because policy, prices, catalog, documents, permissions, user behavior, or external tools changed. Tie alerts to meaningful shifts and observed outcomes rather than a universal drift threshold.

Feed confirmed failures and corrections into a triage queue. Decide whether each becomes a regression case, a new slice, a deterministic control, a product change, or an accepted limitation.

Retain the production decision record for the period required by the task and policy. It should connect the user or system request, evidence version, recommendation, approval or automated policy decision, action receipt, final outcome, correction, and incident reference. Restrict access and avoid retaining raw prompts when a smaller structured record is sufficient.

Failure, rollback, and recovery

Stop or reduce autonomy when a hard safety gate fails, a critical slice degrades, reviewer capacity is exceeded, evidence sources are unavailable, the system version is unknown, or monitoring cannot observe the protected outcome.

Rollback the model, prompt, retrieval, policy, tool, or routing bundle as one governed release unit. A model rollback alone may not restore behavior if the corpus, tool schema, or policy changed.

Reconcile actions already attempted. Identify affected cases, external effects, user communications, and corrections. Preserve evidence for incident review and notify the appropriate owner under the organization’s policy.

Do not add the failed production cases to training or evaluation automatically. Review rights, privacy, representativeness, and whether inclusion would expose the held-out set.

Workbook completion checklist

"The task, affected actor, consequence, evidence boundary, action boundary, and baseline are explicit.", "Unacceptable outcomes, abstention, escalation, expiry, recovery, and allowed autonomy are defined.", "The case set has version, provenance, access, retention, slice labels, and held-out cases.", "Routine, edge, ambiguous, denied, adversarial, dependency-failure, and recovery cases are represented.", "The rubric uses observable criteria and calibrated positive, negative, and borderline examples.", "The complete system version includes model, prompt, retrieval, tools, policy, validators, and configuration.", "Deterministic security and state controls pass independently of model-quality scoring.", "Candidate runs retain outputs, evidence, actions, latency, cost, reviewer judgement, and disagreement.", "Critical slices and unacceptable outcomes have gates that an overall average cannot override.", "The bounded release has permissions, cohort, observation window, stop signals, fallback, and owner.", "Production corrections and incidents enter a governed triage and evaluation-update process.", "Rollback and reconciliation cover the whole release bundle and already-attempted actions." ]} />

An evaluation does not certify a model in the abstract. It supports a bounded decision about one system version, task, population, operating context, and level of authority.

Operating cadence

Run the relevant gate for every material change. Review the full case inventory on a fixed cadence and after incidents, policy changes, new user groups, new tools, new data sources, provider updates, or shifts in production input.

Keep old reports reproducible. Do not overwrite a failing result after a fix. Link the new candidate to the previous failure and show which evidence changed.

The next action is to choose one real task and complete the task contract, slice inventory, and first ten cases before selecting the score the team wants to optimize.