AI Agent Architecture in Production: Control, Evaluation, and Operations

A production architecture and decision framework for bounded AI agents, covering use-case selection, autonomy, identity, tools, approvals, prompt injection, memory,...

audience="CTOs, AI and platform leaders, product owners, security teams and engineers deciding whether an agent can safely perform a real business workflow." decision="Which parts of the workflow may be model-directed, which tools and data may be exposed, where approval is required, and what evidence permits release and continued operation." position="Production agents should be designed as bounded decision systems. The model may propose and adapt, but identity, authorization, action limits, approvals, execution and evidence remain explicit system responsibilities." scope="This paper covers agents that can select tools or affect external state. It is model and framework neutral, and uses illustrative scenarios rather than universal accuracy, savings or recovery claims." outputs={[ 'An agent suitability assessment', 'An autonomy and approval model', 'A tool-risk register', 'An evaluation and release plan', 'An observability and incident design', 'A production-readiness checklist', ]} />

Abstract

An AI agent combines a model with instructions, tools and a loop that can select the next action based on the current state. That ability is useful when work contains ambiguity, unstructured information and exceptions that resist a fixed workflow. It also moves risk beyond the quality of generated text. An incorrect agent can query the wrong data, expose information, create records, send messages, spend money or change a production system.

This paper presents a system architecture for agents that act in real workflows. It begins with the decision of whether an agent is appropriate at all, then defines the boundaries around identity, context, retrieval, tools, memory, approvals, budgets, execution, observability, evaluation and recovery. It treats the model as one component inside a controlled system rather than the security or workflow authority.

The central recommendation is to grant autonomy in small, measurable increments. Start with a bounded task and read-only tools. Establish an evaluation baseline, make action policy independent from the model, preserve full execution evidence and add write authority only when failure behavior is understood. Multi-agent coordination and broad autonomy are later design choices, not the starting architecture.

Definitions and assumptions

| Term | Meaning in this paper | |---|---| | Agent | A system in which a model helps control workflow execution and can select tools or next steps toward a goal. | | Tool | A typed interface to data or an external action. A tool can read, calculate or change state. | | Run | One bounded attempt to complete a goal, with an identity, policy version, state and budget. | | Action budget | Limits on steps, time, tokens, cost, tool calls or consequence. | | Guardrail | A control that detects or prevents a defined class of input, output, policy or action failure. | | Approval | A recorded human decision authorizing a specific proposed action for a limited scope and time. | | Receipt | Immutable evidence of a tool request, authorization decision, external result and verification. | | Evaluation | A repeatable test of the complete agent system, including model, prompts, tools, policies, budgets and runtime. | | Handoff | Transfer of the task and accumulated evidence to a person or deterministic workflow. |

This paper assumes the agent will operate inside an existing identity, data and application environment. It does not assume that one model, framework or cloud service is correct for every workflow. Model behavior and vendor features change, so version-sensitive implementation details must be verified against current primary documentation.

Decide whether to build an agent

An agent is not the default solution to every workflow. A deterministic service is easier to test when rules are stable and inputs are structured. A search or summarization feature may need retrieval and generation without agent-controlled actions. A human may remain the correct decision-maker when consequences are high and judgment cannot be reliably evaluated.

The OpenAI practical guide to building agents recommends agents for workflows with complex decisions, difficult-to-maintain rules or substantial unstructured data, and advises using deterministic approaches when they are sufficient.

Suitability assessment

Score the candidate workflow using actual cases. A high total is not automatic approval. Any unacceptable consequence or missing control can stop the project.

| Dimension | Questions | Evidence | |---|---|---| | Ambiguity | Does the task require interpreting unstructured information or exceptions? | Representative task set and current human decision notes. | | Determinism | Can ordinary rules complete most cases reliably? | Existing rules, decision tree and exception rate. | | Verification | Can the outcome or action be checked before harm occurs? | Deterministic validation, independent source or reviewer rubric. | | Reversibility | Can a wrong action be undone completely and quickly? | Tested rollback or compensating transaction. | | Consequence | What is the worst credible effect on a person, customer, system or account? | Threat model and business impact review. | | Data exposure | Which sensitive data, secrets or tenant boundaries enter the context? | Data-flow and access review. | | Tool authority | Which systems can the agent read or change? | Tool inventory and least-privilege scopes. | | Evaluation | Can success, safety and policy compliance be measured on realistic cases? | Golden cases, adversarial cases and review capacity. | | Economics | Is the expected value greater than model, tool, review and operating cost? | Baseline time, volume, unit cost and exception handling. |

Reject or narrow the use case when success cannot be defined, harmful actions cannot be contained, required data access is too broad, or there is no accountable owner for exceptions.

Worked scenario: a bounded operations agent

The scenario is illustrative. It demonstrates architecture decisions without claiming a client result.

A B2B software company wants to reduce the time required to investigate customer integration failures. Today an engineer opens a ticket, reads logs, checks deployment history, compares the customer configuration, runs a small set of diagnostics and writes a recommended action. Some fixes are safe and repeatable. Others change production configuration or resend business events.

The first release does not allow the agent to modify production. It can:

read the ticket and approved customer metadata;
retrieve logs for the affected tenant and time window;
read deployment and configuration history;
run pre-approved read-only diagnostics;
classify the likely failure and cite supporting evidence;
draft a remediation plan;
hand the plan to an engineer.

The next release may allow two reversible actions: retrying an idempotent diagnostic job and creating a draft configuration change. Sending the change to production remains behind named approval and an ordinary deployment pipeline.

Scenario control table

| Action | Agent authority | Required control | Verification | |---|---|---|---| | Read ticket | Automatic | Ticket scope and tenant binding | Ticket ID appears in the run trace. | | Query logs | Automatic within limits | Tenant, time-range and field filters | Returned log references remain inside the approved tenant. | | Read deployment history | Automatic | Environment and service allowlist | Version and timestamp are cited in the conclusion. | | Run diagnostic | Automatic for allowlisted checks | Runtime, spend and concurrency budget | Typed result and exit status are recorded. | | Retry idempotent job | Approval during early release | Idempotency key and job-state check | External job receipt and resulting state are reconciled. | | Draft configuration patch | Automatic draft only | Schema validation and diff limit | Human reviews exact diff. | | Apply production change | No direct authority | Existing deployment approval and rollback | Deployment system records actor, change, tests and result. | | Notify customer | Draft only | Approved communication template and human review | Sent message is linked to the ticket after approval. |

The architecture is valuable before the agent writes anything. It establishes which actions can be automated, what evidence is required and where the existing production process remains authoritative.

Reference architecture and trust boundaries

The production architecture must separate the online decision path from the evidence and release path. The important boundaries are:

The identity boundary establishes who requested the run, which tenant or account is in scope and why the action is allowed.
The context boundary separates trusted instructions and policy from untrusted user, retrieved and tool content.
The tool boundary prevents a model output from becoming an external side effect without independent validation and authorization.
The evidence boundary records enough state to investigate, evaluate, reconcile and recover.
The release boundary decides which version may serve which cohort with which tools and budgets.

Run envelope

Every run should carry an immutable envelope created by trusted application code:

| Field | Purpose | |---|---| | run_id | Joins model, retrieval, tool, approval and result events. | | principal_id | Identifies the user or workload that initiated the run. | | tenant_id | Binds every data and action path to the active tenant where applicable. | | purpose | Records the allowed business objective. | | policy_version | Identifies the authorization and guardrail rules evaluated. | | agent_version | Identifies instructions, model settings, tools and orchestration code. | | action_budget | Sets limits on steps, tokens, time, cost and tool risk. | | data_scope | Limits retrieval and tool access to approved resources. | | correlation_id | Connects the run to the surrounding product or support workflow. | | expires_at | Prevents a stale run or approval from acting later. |

Do not ask the model to preserve these controls in prose. The runtime passes them outside the prompt and rejects calls that conflict with them.

Workflow design and state

A production agent needs an explicit state machine even when the model proposes the next step. Useful states include requested, context resolved, planning, awaiting approval, executing, verifying, completed, handed off, failed and cancelled.

Each transition should define:

the allowed actor;
preconditions;
state written before and after the action;
timeout and retry policy;
idempotency behavior;
cancellation behavior;
evidence retained;
terminal outcome.

A loop with only “keep calling the model until done” cannot distinguish progress from repetition. Require an exit condition such as a typed final result, a verified business state, an explicit handoff or a hard budget limit.

Planning patterns

Use the simplest orchestration that fits the task:

| Pattern | Fits when | Main risk | Control | |---|---|---|---| | Single tool call | One bounded decision selects one action | Incorrect arguments | Typed schema and deterministic validation. | | Fixed workflow with model steps | Sequence is stable but interpretation is ambiguous | Model output breaks later step | Validate every boundary and retain deterministic orchestration. | | Model-directed loop | The next step depends on observations | Repetition, cost or unsafe exploration | Step, tool, time and spend budgets plus explicit exit states. | | Planner and executor | A plan benefits from review before execution | Plan becomes stale or overly broad | Revalidate each action against current state and policy. | | Multi-agent handoff | Distinct contexts or permissions genuinely require separation | Coordination and evaluation complexity | Typed handoff contract and one accountable orchestrator. |

Prompt templates are easier to evaluate than a collection of tenant-specific prompts. Put variable policy and context into typed fields. Version the template, tool definitions and supporting examples together.

Tool boundary and authorization

The model proposes a tool and arguments. It does not authorize the call. A policy service or deterministic runtime verifies the principal, tenant, purpose, tool risk, argument limits, current state and required approval.

Tool contract

Every tool needs:

a narrow responsibility and unambiguous name;
typed input and output schemas;
authenticated workload identity;
least-privilege scope;
validation independent from the model;
timeout and bounded response size;
retry and idempotency behavior;
side-effect classification;
observable request, result and external receipt;
a test double for evaluation.

Avoid generic tools such as run_sql, call_api, execute_shell or send_request when a narrower interface is possible. A tool named create_refund_draft(order_id, amount, reason) is easier to constrain and evaluate than arbitrary access to a payments API.

Tool-risk register

| Risk class | Examples | Default behavior | |---|---|---| | Read only, low sensitivity | Public documentation, approved product catalogue | Automatic with rate and size limits. | | Read only, sensitive | Customer records, internal logs, contracts | Tenant and purpose binding, field filters and audit. | | Reversible write | Create draft, add internal label, schedule pending job | Automatic only after strong verification; approval during early rollout. | | Consequential write | Send message, change configuration, update CRM status | Specific approval, idempotency and external verification. | | Irreversible or high stakes | Payment, account deletion, legal or medical decision, production credential change | Human-owned decision or separate high-assurance workflow. |

Approval applies to a specific action and arguments, not to a general conversation. Record approver, reason, scope, policy version, expiry and the exact proposed side effect. Re-evaluate if the external state changes before execution.

Defense in depth

No single guardrail secures an agent. Input classifiers may reduce common injection attempts but cannot replace tool authorization. Output validation may catch malformed content but cannot undo an unauthorized call. Use controls that fail independently.

Identity and tenancy

Resolve identity and tenant context before retrieval or planning. Every subsequent read and tool call must enforce the same scope. Do not allow a model to change tenant by generating a different identifier.

Service accounts used by tools should be scoped to the action, environment and data needed. A shared administrative credential turns every prompt-injection or tool-selection failure into a high-impact security event.

Untrusted content and prompt injection

Treat user messages, webpages, documents, emails, tickets, retrieved passages and tool output as data. They may contain text that asks the agent to ignore instructions, expose secrets or call a tool.

Controls include:

mark trusted instructions and untrusted content separately in the runtime;
restrict retrieval by identity, tenant and purpose before content enters context;
minimize secrets and sensitive data in prompts;
prevent retrieved text from creating new tool permissions;
validate tool choice and arguments outside the model;
require approval for higher-risk actions;
scan output for prohibited disclosure before returning it;
test indirect injection through every content source the agent can read.

The OWASP guidance for securing agentic applications emphasizes least privilege, human oversight for high-risk actions and controls around agent tools and data. Use its current threat guidance as an input to the product-specific threat model.

Action budgets

Set hard limits for:

model turns and total tokens;
wall-clock duration;
tool calls by type;
records or files read;
external writes;
concurrent subtasks;
monetary spend;
retries and repeated failures.

The runtime stops or hands off when a budget is exhausted. The model cannot extend its own budget.

Output and side-effect verification

Validate structured output against schema and business rules. For actions, query the external source of truth after execution. A successful HTTP response is not proof that the intended business state exists.

Examples:

confirm the scheduled job exists once with the expected parameters;
confirm the CRM record contains the approved change and no unrelated fields changed;
compare a generated configuration patch against the allowlist and requested scope;
verify the notification provider accepted one message to the approved recipient;
reconcile a payment action against the transaction ledger.

Retrieval, memory and state

Retrieval and memory solve different problems. Retrieval selects current knowledge from approved sources. State records the current workflow. Memory retains information for reuse across turns or runs. Combining them in one unbounded vector store creates unclear ownership and retention.

Retrieval contract

Record source identity, version, access decision, retrieval query, returned references and ranking metadata. Preserve citations into the final answer when a reviewer needs to verify the evidence.

Use hybrid or structured retrieval when exact identifiers, dates or policy clauses matter. Similarity alone can return text that sounds relevant but belongs to another product, tenant or policy version.

Workflow state

Persist business state outside the model context. The state store should record current step, completed actions, pending approvals, tool receipts, budgets and terminal outcome. A run can then resume without asking the model to reconstruct history from a long transcript.

Memory policy

For each memory type, define:

| Question | Example decision | |---|---| | What is retained? | User-approved preference, not every conversation sentence. | | Why is it retained? | Reduce repeated setup for a named workflow. | | Who may read it? | The same user and tenant for the same purpose. | | How long? | A documented TTL aligned with product need. | | How is it corrected? | User or operator can inspect and remove inaccurate state. | | How is it evaluated? | Tests detect stale, cross-tenant or irrelevant recall. |

Do not store model-generated assumptions as facts. Keep provenance and confidence when derived information must persist.

Evaluation and release evidence

An evaluation measures the whole tested system. Report the model, reasoning settings, prompts, tool access, policies, budgets, data and scoring method. A model score from a different harness does not predict the deployed agent’s behavior.

The NIST Generative AI Profile frames risk management across the AI lifecycle. Use it with the organization’s existing security, privacy and model-governance processes rather than treating agent evaluation as a standalone benchmark.

Evaluation set design

Build several connected sets:

common successful tasks;
difficult but valid tasks;
ambiguous requests that require clarification;
unsupported requests that require refusal or handoff;
stale, missing and conflicting data;
tool timeouts and partial failures;
direct and indirect prompt injection;
cross-tenant and unauthorized-resource attempts;
high-cost loops and repeated retries;
known production failures and user corrections.

Store expected policy and outcome, not only one expected sentence. Several outputs can be acceptable if they use the right evidence, follow policy and reach the correct business state.

Evaluation dimensions

| Dimension | Example measure | |---|---| | Task outcome | Correct business state or accepted handoff. | | Evidence | Required sources cited and no unsupported source used. | | Policy | Allowed tools, data scope, approvals and refusal behavior. | | Safety | No prohibited disclosure or action in adversarial cases. | | Efficiency | Steps, tokens, tool calls, latency and cost per accepted outcome. | | Recovery | Correct response to timeout, stale state and partial side effect. | | User experience | Clarification quality, progress visibility and useful handoff package. |

Combine deterministic checks, model-assisted grading and human review. Deterministic checks fit schemas, permissions, citations, tool sequences and external outcomes. Model-assisted graders can assess semantic quality but need calibration and spot review. Human review is essential for ambiguous quality, policy interpretation and new failure classes.

Release stages

Offline evaluation with test doubles for external tools.
Integration evaluation in an isolated environment.
Shadow mode on representative traffic without external side effects.
Internal or low-risk cohort with read-only tools.
Canary with bounded reversible actions and approval.
Controlled expansion by user, tenant, workflow and tool class.
Continuous evaluation and rollback based on production evidence.

Define release and rollback thresholds before the canary. Avoid selecting only easy traffic that hides the workflows the system must eventually serve.

Observability

Tracing should let an operator reconstruct what the system knew, decided, attempted and changed without logging hidden reasoning or unnecessary sensitive content.

Capture:

run, principal, tenant, purpose and agent version;
model request metadata and response type;
retrieval sources and access decisions;
tool name, validated arguments, policy result and approval;
tool latency, status and receipt;
step, token, time and spend budgets;
final outcome, handoff or stop reason;
user correction, override and reported defect;
release cohort and policy version.

Redact secrets, credentials and unnecessary personal data. Use references or hashes where operators need correlation without full content. Restrict trace access because traces may contain more context than ordinary application logs.

Operational views

Create views for:

outcome acceptance and handoff rate by workflow;
policy denial and approval rate by tool;
repeated loops, retries and budget exhaustion;
latency and cost by accepted outcome, not only per model call;
retrieval misses and unsupported citations;
user corrections and reopened tasks;
tool and dependency failure;
safety events and unauthorized attempts;
drift by agent version and release cohort.

A low error rate can hide a bad agent if users stop trusting it and complete the work manually. Measure workflow completion and correction, not just API success.

Runtime ownership, checkpoints, and uncertain outcomes

Externalizing run state does not by itself prevent two workers from continuing the same run. Give each active run a time-bounded lease and monotonically increasing fencing token. A worker may commit a checkpoint or invoke a state-changing tool only when its lease and token are still current. If ownership changes after a timeout, the replacement worker reads the last atomic checkpoint and rejects results produced under an older token.

A checkpoint should commit the workflow state, pending action, policy version, budget consumed, approval reference, and last verified external receipt together. Do not mark an action complete merely because the request was sent. Record one of three states: confirmed, rejected, or uncertain. An uncertain write enters reconciliation. The system queries the external source of truth, matches the idempotency key or provider receipt, and only then decides whether to continue, compensate, or request human review.

Bind an approval to the run, proposed action, validated arguments, policy version, approver, consequence class, and expiry. Any material change after approval invalidates it. A broad approval for one run must not authorize a later retry, another tenant, or a modified payload.

Failure containment and incident response

Contain side effects before diagnosing the model. A kill switch should disable a tool, agent version, tenant cohort or action class without taking the entire application offline.

Failure classes

| Class | Example | Immediate containment | |---|---|---| | Data | Retrieved stale policy or wrong tenant document | Block further use of the source and invalidate affected runs. | | Model | Repeatedly selects an irrelevant plan | Stop the run, preserve trace and hand off. | | Tool | Partial write or ambiguous timeout | Prevent retry until external state is reconciled. | | Policy | Approval bypass or incorrect scope | Disable the action path and review all affected receipts. | | Dependency | Identity, retrieval or external API unavailable | Fail closed for writes; use defined degraded behavior for reads. | | Cost | Loop or fan-out exceeds budget | Cancel pending work and retain the last safe state. | | Security | Injection, data exfiltration or privilege misuse | Revoke credentials, isolate the run and start security response. |

Retry policy

Retry only when:

the error is classified as transient;
the operation is read-only or idempotent;
external state is known;
the retry remains inside the original authorization and approval;
the budget permits it;
the retry will be visible in the same run trace.

Never ask the model to decide whether an ambiguous financial, messaging or configuration write is safe to repeat.

Incident evidence

Preserve the agent version, policy, prompts, retrieved references, tool definitions, validated arguments, approvals, receipts, external state and user-visible result. Convert the failure into an evaluation case only after the expected policy and outcome have been reviewed.

Scaling and cost

Agent cost includes more than tokens:

cost per accepted outcome =
  model calls and reasoning
  + retrieval and reranking
  + tool and external API usage
  + storage, tracing and evaluation
  + approval and exception handling
  + retries and failed runs
  + engineering and on-call operation

Track cost by workflow and outcome. A cheaper model can increase total cost if it makes more tool calls, needs more retries or creates more human review. A larger model can be economical on a narrow difficult decision and wasteful on deterministic classification.

Routing strategy

Start with the most capable model needed to establish a quality baseline. Then test smaller or faster models on bounded subtasks. Route by evaluated task class, not by prompt length alone.

Deterministic code should handle validation, policy, arithmetic, identifier lookup and state transitions whenever possible. Reserve model inference for interpretation, planning and generation where it adds value.

Concurrency and backpressure

Bound concurrent runs by tenant, workflow and tool. Queue expensive research or batch work separately from interactive tasks. Propagate cancellation. A user who closes or cancels a task should not leave a costly agent running and later writing state.

Multi-agent systems

Multiple agents are appropriate when distinct contexts, permissions or ownership make a separation valuable. Examples include a researcher with read-only access handing a cited brief to an action agent, or specialist agents coordinated by one accountable workflow.

Do not split the system because roles sound organizationally attractive. Each additional agent creates another prompt, context, tool boundary, failure path and evaluation surface.

Require:

a typed handoff object;
clear ownership of the final decision;
independent permissions;
bounded fan-out and recursion;
one trace across the system;
tests for partial and conflicting results;
a stop condition when agents disagree.

A single agent with well-designed tools is often easier to operate than a collection of agents that summarize one another.

Limitations and non-goals

This architecture cannot make a model deterministic, guarantee factual output or remove the need for product-specific risk analysis. It does not replace ordinary application security, access control, privacy review, change management or incident response.

Avoid or sharply limit agent action when:

consequences are irreversible or affect a person’s rights, health, employment, credit or safety;
the required data cannot be scoped without broad privileged access;
reviewers cannot determine whether an outcome is correct;
the workflow has low volume and high exception cost;
a deterministic service can perform the task more reliably;
the organization cannot preserve and review execution evidence;
external systems cannot support idempotency, reconciliation or rollback.

The correct product may be AI-assisted search, drafting or decision support with a human-controlled action.

Production-readiness workbook

Record evidence and an owner for every “no” or “unknown.”

Use case and autonomy

[ ] Representative tasks and exceptions have been sampled from the real workflow.
[ ] Deterministic alternatives were evaluated.
[ ] Success, refusal and handoff are defined as business outcomes.
[ ] Consequence, reversibility and verification determine the autonomy level.
[ ] An accountable product and operational owner are named.

Identity, data and memory

[ ] Principal, tenant and purpose are established outside the model.
[ ] Retrieval enforces identity and scope before content enters context.
[ ] Secrets and unnecessary sensitive data are excluded from prompts and traces.
[ ] Workflow state has an authoritative store outside model context.
[ ] Memory has purpose, provenance, access, correction and retention rules.

Tools and actions

[ ] Every tool has typed schemas, least privilege, timeout and observable receipts.
[ ] The model cannot authorize its own tool call or extend its own budget.
[ ] Approval binds to exact action, arguments, scope and expiry.
[ ] Writes are idempotent or have an explicit reconciliation path.
[ ] High-consequence actions remain human-owned or in a separate high-assurance workflow.

Evaluation and release

[ ] The evaluation set includes real, difficult, ambiguous, adversarial and failure cases.
[ ] The tested system version includes model, prompts, tools, policies and budgets.
[ ] Deterministic checks, calibrated graders and human review are assigned appropriately.
[ ] Shadow, canary, expansion and rollback criteria are defined.
[ ] Production failures and user corrections become reviewed regression cases.

Operations and recovery

[ ] A trace joins the request, evidence, decisions, tools, approvals and external outcome.
[ ] Kill switches can disable one tool, version, cohort or action class.
[ ] Partial side effects are reconciled before retry.
[ ] On-call can distinguish data, model, tool, policy, dependency and cost failures.
[ ] Cost is measured per accepted outcome, including review and failed runs.

What to do Monday morning

Select one workflow with clear volume, pain and an accountable owner.
Collect 30 to 50 representative cases, including exceptions and failures.
Draw the data and action boundary, including every system the agent would read or change.
Classify each proposed tool by sensitivity, reversibility and consequence.
Define the first release with read-only tools and explicit handoff.
Write the expected policy and outcome for ten cases before building prompts.
Build the run envelope, trace and evaluation harness before adding more autonomy.

Primary references

Use these references with current model, framework and cloud documentation. Validate permissions, retention, regional behavior, tool semantics and safety controls against the exact deployed versions.