AI Agent Development Playbook for Production Systems

Build a bounded agent workflow with independently authorized tools, durable action records, representative evaluations, budget controls and tested recovery.

Start with a task the team can accept or reject

An agent can choose a next step using model output, call tools and continue from their results. That flexibility adds failure modes: repeated actions, incorrect tool selection, misleading retrieved instructions and runs whose outcome is uncertain. Use it when the task requires that flexibility, not because an agent framework makes it easy to build a demo.

This playbook takes one bounded workflow from a use-case decision to a controlled production release. It covers tool-using agents that propose or perform business actions. For a read-only assistant, keep the same data-access and evaluation controls but omit write paths that do not exist. Use the production-grade AI systems playbook for the broader service architecture and release controls.

The product owner starts with a deterministic baseline. A fixed sequence of retrieval, validation and API calls may satisfy the task with clearer behavior. Compare completion quality, review effort, elapsed time, accepted cost and failure recovery. A model's ability to produce a plausible answer is not evidence that it can complete the business workflow.

1. Write the use-case contract

Choose one workflow and one user population. “Handle support” is too broad. An illustrative bounded task is to collect evidence for a support request and propose the next permitted step. Sending the response or issuing a refund is a separate authority decision.

Use this record before selecting a model or orchestration library:

Workflow / accountable product owner:
Eligible users, tenants and inputs:
Successful outcome and authoritative evidence:
Allowed reads / allowed writes / prohibited actions:
When the workflow must ask for clarification:
When a human must take over:
Maximum elapsed time, spend and outstanding actions:
Fallback when model, tool or reviewer is unavailable:
Baseline workflow and comparison criteria:
Data retention and deletion owner:

Define ambiguous cases explicitly. A missing account ID should cause clarification or handoff, not a guessed customer lookup. An unavailable reviewer should leave a pending proposal or return a clear fallback, not convert an approval requirement into automatic execution.

Gate: the product and security owners can review sample tasks and agree which outcomes are acceptable. If they cannot, refine the task before implementing an agent loop.

2. Put authority outside the model

Treat a tool call as a proposal. The execution service derives identity and tenant scope from the authenticated session, validates the requested action and checks current authorization. Do not trust a tenant ID, role or approval flag supplied by the model.

The tool/API owner implements checks at the service that performs the effect. A prompt saying “never access another tenant” cannot replace row-level or resource-level authorization. Restrict network destinations and tool scopes. Keep production credentials out of prompts and tool responses.

Approval should bind to the exact normalized action, affected resource, actor, policy version and expiry. If the proposal changes after review, require new approval. Revalidate authorization at execution time because an account, role or resource may have changed while waiting.

OWASP's prompt-injection guidance describes tool-specific validation and least-privilege access as controls alongside model-level defenses. Delimiters and a second model can help with behavior, but neither establishes authorization.

3. Design tools around recoverable business operations

Give each tool a clear purpose and a bounded contract. Split read and write capabilities when their permission or recovery requirements differ. A combined operation can still be appropriate when the backend provides an atomic business transaction. Do not split operations merely to increase the number of tools.

For effectful operations, idempotency means that retrying the same intended operation does not create additional effects within the contract's scope. It does not mean every call with similar inputs should be collapsed. Two separately authorized purchases can have identical amounts.

Persist a stable operation ID before sending a write. Reuse it for the same intent, reject conflicting payload reuse, and keep the result long enough for the expected retry window. The AWS Builders' Library discussion of idempotent APIs explains why caller-provided identity and intent matching matter.

Gate: contract tests demonstrate duplicate delivery, concurrent calls and a response lost after the effect. If the downstream system offers no idempotency or reliable lookup, document the remaining duplicate-action risk and require an appropriate manual path.

4. Model durable run state and bounded memory

Store run state outside the model context. Record the workflow version, action proposals, approval references, completed action receipts, pending work and stop reason. A worker restart should reload that state before deciding what to do next.

Conversation history is not the authoritative action record. A model summary may omit a prior effect or change its meaning. Keep accepted business facts and action receipts separate from generated narrative.

Use memory deliberately. Session context, retrieved organizational data and saved user preferences have different permissions and retention needs. Every stored item needs a source, scope and deletion path. Do not turn an unverified model inference into a durable customer fact. Check access again when retrieving an older record, and invalidate caches when permissions or source data change.

5. Test deterministic controls and model behavior separately

Traditional unit and integration tests remain useful for schema validation, authorization, state transitions, idempotency, budget admission and recovery. Model evaluations cover whether the system chooses appropriate steps and produces acceptable outcomes across representative inputs.

The evaluation owner builds a versioned set from permitted examples. Include routine tasks, ambiguous requests, denied actions, cross-tenant attempts, conflicting evidence, long conversations, malicious tool output and unavailable dependencies. Cover the workflow's risk categories rather than selecting an arbitrary number of cases.

Evaluate the complete trajectory. A correct final answer can follow an unauthorized read or a duplicate write. Record outcome correctness, action legality, clarification quality, escalation behavior, unnecessary work, time and cost. Fewer tool calls are useful only when the accepted outcome and safety requirements remain satisfied.

Separate development examples from a held-out evaluation set. Use human review for disputed or consequential judgments, with a written rubric and disagreement process. A model judge can assist triage, but its judgment is not independent proof.

Gate: deterministic controls pass and the product owner accepts measured tradeoffs against the baseline. There is no assumed monthly improvement rate. New traces create candidates for investigation; they do not automatically justify retraining or prompt changes.

6. Admit work before spending the budget

Track elapsed time, model calls, tool attempts, concurrency and monetary exposure at run and tenant level. Checking a cumulative total only after a call returns cannot enforce a hard pre-call limit.

Before dispatch, reserve an upper-bound allowance using the configured request limits and dated rate assumptions. Make the reservation atomic across concurrent workers. If no valid allowance remains, stop, defer or escalate. After the call, settle observed usage and release any unused reservation. Include retries, tool charges and reviewer cost in the appropriate budget.

An upper bound may be uncertain because providers, tools or exchange rates differ. State that limitation and keep a safety margin approved by the owner. Use precise monetary units and a shared ledger rather than a floating-point counter local to one worker.

A budget stop must not erase an in-flight effect. Stop new actions, save the run state and reconcile outstanding writes. Explain to the user whether the task is complete, partial, awaiting review or stopped. “Budget exceeded” alone does not describe what happened to their request.

7. Release behind a recoverable boundary

The service owner begins with a read-only or proposal-only cohort when the use case permits it. Keep consequential actions disabled until approval, idempotency and reconciliation tests pass. Select cohort size and observation conditions from risk and traffic, not a universal rollout percentage.

Pin the prompt, model configuration, tool contract, policy and evaluation dataset used for the release. Record provider/model identifiers and known versioning limits. Re-run relevant tests when any of these dependencies changes.

8. Test the agent as a stateful system

A single-turn answer test cannot prove an agent workflow. Evaluate complete trajectories, including what the system read, proposed, authorized, executed, observed and retained. Two runs with the same final sentence can have very different safety and cost profiles if one attempted an unauthorized read or repeated a write.

Build scenario families rather than isolated prompts:

Retain the workflow version, starting state, tool fixtures, expected state transitions, actual trajectory, action receipts and reviewer judgment. Keep production side effects disabled in evaluation unless a separately authorized exercise requires them.

Test whether the agent stops. Give it an impossible task, contradictory evidence, a permanently unavailable dependency and a tool that repeatedly returns incomplete information. The accepted result should be a clear partial outcome or escalation, not an expensive loop that eventually times out.

9. Operate with a queue and action ledger

Operators need separate views of runs, proposed actions, approvals, in-flight effects and unknown outcomes. A single conversational transcript cannot show which records require reconciliation or which approvals are about to expire.

Prioritize the unknown-outcome queue by consequence and age. For each item, show the stable operation ID, intended effect, downstream resource, last request time, available status lookup and person authorized to resolve it. Do not let a general “retry failed runs” control include ambiguous writes.

Set alerts on control failures: growing unknown outcomes, repeated policy denials from one workflow version, approval backlog, budget reservations that never settle, and runs that approach step or time limits. Alerting on every model refusal creates noise; alert when the service needs an operator decision.

Review a sample of successful runs as well as failures. A workflow can appear healthy while using unnecessary tools, retrieving excessive data or requiring silent human correction. Compare real outcomes with the baseline and retire agent steps that add flexibility without decision value.

Retain each operational decision with the release version and affected cohort. This makes later expansion, rollback and incident review depend on evidence rather than recollection.

10. Operate, investigate and recover

Use run IDs to connect model activity, policy decisions and tool receipts. Collect enough telemetry to explain actions without logging every prompt or response by default. Redact credentials and unnecessary personal data, restrict access, define retention and test deletion. If raw samples are necessary, approve their purpose and storage separately. The data owner must also confirm the collection basis, required user notice and any applicable consent requirements before sampling.

Replay recorded tool responses in a no-side-effect test environment. Never replay a production write merely to reproduce a trace. Low temperature can reduce some variation, but it does not guarantee identical output. Preserve inputs, versions and observed decisions so investigators can compare behavior without claiming exact reproduction.

Changing a prompt cannot undo a sent email, payment or data disclosure. Compensation is a new authorized action with its own risks. The incident owner tracks reconciliation independently of the model rollback.

Acceptance checklist and limitations

• The product owner approved eligible tasks, prohibited outcomes and the baseline comparison
• Tool services enforce identity, scope and authorization independently of the model
• Approval binds the exact action and expires safely
• Duplicate, concurrent and unknown-outcome tests have recorded results
• Memory, traces and evaluation examples have access and retention controls
• Budgets reserve before dispatch and stop new work without abandoning reconciliation
• The operator demonstrated the kill switch, fallback and recovery process
• The release record names residual risks and the person accepting them

This playbook does not certify an agent as safe for every task. Multi-agent delegation, regulated decisions and irreversible financial or safety-critical actions need additional domain review. Model behavior can change, test sets can miss attacks, and downstream tools can violate their contracts. Keep factual and security approval separate from an editorially complete implementation guide.

For a scoped implementation discussion, bring the use-case contract and one failed trajectory to agentic workflow engineering. Those records make the review concrete.

Primary references

• OWASP: LLM Prompt Injection Prevention Cheat Sheet, independent controls and agent-specific defenses.
• AWS Builders' Library: Making retries safe with idempotent APIs, operation identity and intent.
• NIST AI 600-1: Generative Artificial Intelligence Profile, risk-management context, not a certification checklist.