LLM Guardrails Playbook for Production Systems

Define enforceable LLM output controls, validate syntax and business meaning, bind approvals to actions, and test safe behavior when a guardrail fails.

trigger="An LLM output will reach a user, another system or a tool with consequences beyond a disposable draft." owner="The service owner owns enforcement and recovery; the product and risk owners approve acceptable outcomes." participants={["AI engineer", "Security reviewer", "Product owner", "Data owner", "Human-review lead", "On-call operator"]} prerequisites={["A mapped data and action flow", "A risk register for the exact use case", "Representative permitted and prohibited examples", "A non-model authorization path and an owned fallback"]} outputs={["Versioned control policy and enforcement map", "Output-validation and approval contracts", "Labeled evaluation results", "Outage, rollback and reconciliation procedures"]} doneWhen={["Rejected or incomplete output cannot reach a protected sink", "Business permissions are enforced independently of the model", "False accepts and false rejects are reviewed by risk category", "Operators have exercised control failure and recovery"]} />

Guard the place where output becomes consequential

An LLM response can be harmless prose in one product and an executable instruction in another. A string containing a database query, customer email or refund request needs controls appropriate to the destination. Start with what the application can do with the output.

This playbook covers the path from model response to display or action. It complements production AI system design and agent development. Its goal is an enforceable contract: which outputs may proceed, which must stop, and who handles uncertainty.

A model can help classify content or flag suspicious instructions. Its judgment can also be wrong. Keep identity, authorization, tenant isolation and critical business rules in application or service controls. Do not treat a second model as a security boundary.

1. Define unacceptable outcomes and the safe fallback

The product owner and security reviewer map each output destination. Record harm severity, reversibility, affected users and available recovery. A generated internal draft and a production database write should not share one allow-or-block rule.

| Destination | Example failure | Required control before release | |---|---|---| | Plain-text answer | Unsupported assertion presented as fact | Evidence policy, uncertainty handling and relevant evaluation | | Rendered web content | Script, unsafe link or active content | Context-appropriate encoding and restricted rendering | | Structured business record | Valid JSON with an invalid account or amount | Schema plus business and authorization checks | | Tool or external API | Unauthorized or repeated action | Scoped executor, approval where required and operation identity | | Sensitive-data response | Cross-tenant or unnecessary personal data | Access checks, purpose limitation and output review |

For each row, select a fallback: clarify, redact, return a supported partial answer, refuse, queue for review or disable that action. A fallback must be understandable to the user and operationally staffed. An endless “pending review” queue is an availability failure.

Gate: the accountable owners approve the risk categories and fallback behavior. Do not start with a universal block-rate target.

2. Map the controls to their enforcement points

Put checks near the protected resource. A UI validation can improve feedback, but the server must reject an invalid action even if the UI is bypassed. Apply tenant scope before retrieval and recheck resource access when executing an action.

"type": "flow", "title": "Checks between a model response and a protected destination", "steps": [ ], "caption": "A failure routes to the use-case fallback. Passing an earlier check does not grant permission to skip later checks or access a protected resource." }} />

The security reviewer records every path to the sink, including retries, streaming output, background workers and human edits. If a streaming answer can leak sensitive content before validation finishes, buffer that output or restrict the feature. A final validator cannot recall tokens already delivered.

Separate policies for content quality, data handling and action authority. This lets an operator disable a failing optional classifier without accidentally removing a critical authorization check.

3. Handle response status before parsing

Provider APIs may report a refusal, incomplete generation, timeout or other error outside the requested application schema. Inspect the documented response status and completion fields before consuming content. Do not repair a refused or truncated action into an executable request.

OpenAI's Structured Outputs documentation distinguishes JSON mode, which targets valid JSON, from schema-constrained output. It also documents refusals and incomplete-response handling. Check the selected provider, model and API version because supported schema features and failure signals differ.

| Check | What it establishes | What it does not establish | |---|---|---| | Successful complete response | The API returned a consumable result under its contract | Factual correctness or permission | | JSON parsing | Syntactically valid JSON | Required fields, allowed values or safe actions | | Schema validation | Conformance to the supported data shape | Truth, account ownership or business acceptability | | Semantic validation | Application rules hold for the proposed values | Authorization unless explicitly checked | | Authorization and approval | This actor may perform this exact action now | Successful execution or absence of downstream failure |

Reject unknown fields where the contract requires a closed schema. Apply length, range and enumeration limits before calling tools. Treat dates, currencies, identifiers and units as typed business data. A syntactically valid amount in the wrong currency can be more damaging than malformed JSON.

Bound repair attempts by time, cost and risk. Repair can be appropriate for a display-only formatting failure. For a consequential action, do not let repeated generation widen permissions or silently change the user's intent.

4. Reduce prompt-injection exposure without promising prevention

User text, retrieved documents, web content and tool output may contain instructions that conflict with the application's task. Mark their provenance and separate them from trusted instructions. Delimiters help the model distinguish content, but an attacker can still influence model behavior.

The independent controls remain:

  • Restrict which tools and network destinations the service can reach.
  • Derive user and tenant scope from trusted session data.
  • Authorize every resource read and write at the destination service.
  • Validate tool arguments against the current task and business policy.
  • Require specific approval for actions whose impact warrants it.
  • Treat retrieved content and tool output as untrusted even after a previous model summarized them.

These controls follow the defense-in-depth approach described in OWASP's prompt-injection guidance. They reduce exposure; they do not establish a universal percentage reduction in attacks.

The evaluation owner includes direct instructions to ignore policy, instructions hidden in retrieved material, encoded and cross-language variants, malicious links, spoofed approvals and multi-turn attempts to expand scope. Test whether the protected sink remains inaccessible, not whether the model says it resisted the attack.

5. Validate claims and sensitive data before display

For evidence-based answers, require source identity and a supporting passage for material claims. Check whether the cited source supports the claim, is current enough for the task and was available to this user. A plausible URL or citation marker is not proof.

For numerical outputs, validate units, arithmetic and the provenance of inputs. For retrieval-based answers, distinguish a failed search from evidence that something does not exist. When evidence is missing or contradictory, return the supported portion, ask a question or hand off. Telling the model to express uncertainty is useful behavior guidance, not a guarantee against fabricated claims.

The data owner defines which sensitive fields are necessary for the task and where they may travel. Apply access controls before model processing. Redaction and output detection are additional layers and can miss formats or create false positives. Do not send confidential data to a model and rely on the output filter to make the earlier disclosure acceptable.

Log policy outcomes and reason codes with minimal content. Raw prompts and responses need an explicit collection purpose, access controls and retention rules. Preserve enough restricted evidence to investigate an incident without copying sensitive material into broad dashboards.

6. Make human approval specific and durable

A reviewer needs the proposed action, affected resources, supporting evidence, uncertainty and likely consequence. A generic “approve agent” button gives too little information.

Bind the decision to a normalized payload, actor, resource scope, policy version and expiry. If the payload changes, approval no longer applies. Recheck authority when executing. Reviewers may reject, edit or ask for more evidence, but an edit must pass the same validators as model output.

"type": "flow", "title": "Approval applies to one action version", "steps": [ ], "caption": "Changed or expired proposals return to review. A timeout after execution enters reconciliation, not an automatic request for a fresh approval and duplicate action." }} />

The review lead sets queue ownership, escalation rules and an availability fallback. Sample accepted and rejected cases for reviewer disagreement. If reviewers lack the evidence or time to evaluate an action, reduce the action scope.

7. Build an evaluation set that exposes control failures

Separate deterministic tests from probabilistic evaluations. Unit and integration tests should cover denied resources, schema failures, approval expiry, duplicate requests and unavailable validators. Model evaluations should cover acceptable completion, unsupported claims, harmful content, injection attempts and overblocking.

Use labeled examples by risk category and user population. Keep a held-out set and document how labels were reviewed. Do not optimize only against known attack strings. Include legitimate inputs that resemble attacks, such as a security analyst discussing malicious prompts.

Report denominators. A false-accept rate uses prohibited cases as its denominator; a false-reject rate uses legitimate cases. Production block rate measures traffic rejected by the policy and is not interchangeable with either. A high block rate can reflect an attack campaign, a product mismatch or an overly broad policy.

The release owner approves thresholds for the specific harm and test conditions. Compare the new policy with the baseline on the same examples, then inspect disagreements. No single aggregate score should hide a cross-tenant disclosure or unauthorized action.

8. Test control outages and recover without bypassing policy

| Failure | Response | Recovery gate | |---|---|---| | Required authorization service unavailable | Stop protected actions; offer the approved limited mode | Authority can be checked again | | Optional quality classifier unavailable | Use the documented degraded path for that risk class | Measured behavior remains acceptable | | Model refusal or incomplete response | Return a clear status or bounded retry when appropriate | A complete permitted result passes validation | | Human-review backlog | Pause new consequential actions or hand off | An accountable reviewer can meet the queue contract | | Bad policy release overblocks users | Restore a known compatible version | Critical enforcement remains intact; regression cases pass | | Unsafe action or disclosure occurred | Contain access and start incident response | Effects and exposure are assessed, not merely filtered later |

A blanket fail-open rule is unsafe. A blanket fail-closed rule can also create an unnecessary outage for low-risk features. Decide the behavior per control and destination before the outage.

Rollback the policy, model or routing configuration only after checking compatibility with stored requests and approvals. Keep already rejected, canceled and completed action identities stable. Reconcile unknown outcomes using the authoritative downstream system. A software rollback cannot unsend a message or erase information already disclosed.

9. Control streaming and partially visible output

Streaming improves perceived latency but changes the enforcement point. Once a token reaches a browser, log, speech synthesizer or downstream parser, a later validator cannot recall it. Decide which destinations may stream and which require a complete buffered result.

For low-consequence prose, the application may apply pre-generation access checks, restricted rendering and incremental content controls, then stop the stream when a policy condition appears. Document that earlier content may already be visible. For sensitive data, regulated advice, code execution or structured actions, buffer the complete output and apply every required check before release.

Test interrupted streams, provider errors after partial content, client reconnects and duplicate consumption. The UI should not present a truncated sentence as a completed answer. If a structured stream cannot be validated atomically, keep it out of the protected action path.

10. Govern policy changes like production code

Version prompts, rules, schemas, classifiers, thresholds, allowlists and reviewer guidance as one control release. Record who changed them, why, which evaluation set passed and which previous version remains recoverable. A model update can alter the distribution of outputs even when the written policy is unchanged.

Use shadow evaluation or replay to compare a candidate policy with the current release. Review both newly blocked legitimate cases and newly allowed prohibited cases. Segment results by destination and risk category because one global threshold can be too strict for drafting and too weak for actions.

Release policy changes progressively where the architecture permits it. Keep critical server-side authorization independent and unchanged during experiments. If the candidate creates an unexpected block or bypass pattern, route affected work to the known policy or safe fallback and preserve the examples for investigation.

Set a review trigger for new tools, new data classes, broader user groups, provider changes and material incident findings. A control approved for internal read-only answers should not silently become the policy for customer communication or external writes.

11. Measure guardrail value and operating cost

Track prevented harmful outcomes, false accepts found in review, legitimate work blocked, time to resolution, reviewer load, latency and cost. Production block rate alone does not tell whether the control is working. A sudden increase may indicate an attack, a bad policy release, a new user behavior or a broken upstream format.

Sample accepted cases according to consequence, not only rejected cases. Otherwise false accepts remain invisible. Give reviewers the evidence and destination context needed to judge the output, and measure disagreement on material cases.

When a control adds substantial latency or review burden, test a narrower enforcement point rather than removing it. For example, a strict approval may apply only when a proposed action exceeds a limit or reaches a protected sink. The residual risk and threshold still require accountable approval.

Document that decision with the affected destination, evidence, approver and next review date. Do not generalize a narrow exception to unrelated outputs.

Reusable control record

Control ID / version / owner:
Protected destination and unacceptable outcome:
Inputs and authoritative identity source:
Enforcement point and all paths that reach it:
Validation rule or classifier configuration:
Allow / deny / clarify / review behavior:
Timeout, dependency outage and degraded-mode behavior:
Approval binding and expiry, if required:
Test set version, labels and risk-specific results:
Telemetry, redaction and retention:
Rollback version and compatibility checks:
Residual risk, approver and next review trigger:

Use one record per independently operated control. Link it to tests and the release record. Keep the reason code understandable enough that support can explain the outcome without exposing internal security details.

Release checklist and limitations

"Every output destination has a named risk owner and fallback", "Refusals, incomplete responses and malformed content cannot become actions", "Schema, semantic and authorization checks are tested separately", "Prompt-injection tests include untrusted retrieved and tool content", "Approvals bind the exact proposal and are revalidated before execution", "Evaluation results separate false accepts, false rejects and production block rate", "Outage behavior and rollback preserve critical controls", "The team has exercised reconciliation for an unknown action outcome" ]} />

Guardrails reduce defined risks under tested conditions. They do not prove factual correctness across all inputs, eliminate prompt injection or establish legal compliance. High-impact decisions need domain-specific review and accountable human judgment. Provider changes, new data sources and new tools can invalidate earlier test evidence.

Use LLM evaluation and observability support when the team needs to turn these control records into repeatable tests and operational evidence.

Primary references