System Architecture: Decisions, Invariants and Operating Evidence

A decision-grade architecture method connecting business invariants to transaction boundaries, derived data, failure behavior, capacity and operational ownership.

Decision brief

Audience: technical leaders and staff engineers deciding how a production product should store state, divide responsibility and behave under failure.

Decision: which boundaries are necessary to protect the product's invariants, and which additional components earn their operating cost.

Thesis: architecture is a set of explicit trade-offs backed by observable behavior. A technology category, company anecdote or diagram cannot establish correctness, resilience or useful capacity.

The worked inventory scenario below is illustrative. It is not a customer architecture, measured performance result or account of Ampity delivery. Its artifacts are an operation contract, a cache-race fixture, a capacity worksheet and an architecture decision record.

This paper owns system-level boundaries. The communication paper goes deeper into delivery, deduplication and compensation after those boundaries are chosen. A team can use this method without adopting microservices, a service mesh or a particular cloud.

1. Define the operation that must remain correct

Suppose a product displays approximate inventory availability while allowing customers to reserve units. Browsing can tolerate a stated amount of staleness. A reservation cannot be approved solely from a stale display if the product promises not to allocate the same available unit twice.

That distinction creates two operation contracts. A browse response may be derived and delayed. A reservation must consult an authoritative state transition with a concurrency-control strategy. The requirements, not the label “read-heavy,” determine where caching is acceptable.

Write the invariant in a form that can fail a test. For the teaching scenario: committed reservations must not reduce available quantity below zero, and repeating the same accepted request identity must not reserve additional units. This excludes returns, expiry and multiple warehouses until their semantics are separately defined.

Ask who can change the rule. Product owns customer-visible promises; the data owner owns consistency assumptions; security owns authorization requirements; operators own recovery execution. Architecture review should expose disagreements among these owners before implementing a topology.

Define acceptance, completion and rejection separately

The reservation contract needs more than a successful HTTP response. Acceptance means the system has taken responsibility for a named request under an agreed deadline. Completion means the authoritative reservation exists and the client can retrieve its outcome. A rejected request has not acquired inventory. A request whose response was lost can be complete on the server while still uncertain to the client.

Give each logical reservation a stable identity scoped to the authorized actor or tenant. Record the material request parameters with that identity. Reusing the identity with a different product or quantity must produce an explicit conflict, not silently replay an unrelated result. Define retention of request identities from the possible retry and recovery window rather than selecting a convenient cache expiry.

Product must also define whether a reservation expires, whether it can be renewed, and what happens if payment arrives after expiry. Those rules introduce competing transitions. An expiry worker and a confirmation request cannot both treat the same quantity as available under different assumptions. Include the transition version or another concurrency mechanism in the authoritative operation, and test the race.

The output of this discussion is an operation contract containing identity, authorized inputs, preconditions, allowed state changes, terminal results, deadlines, and the permitted response to uncertainty. Keep implementation choices outside the contract until the behavior is agreed. Otherwise a tool's convenient default can become an accidental customer promise that later components cannot honor.

2. Record constraints and uncertainty

Use a compact constraint ledger:

| Constraint | Decision input | Evidence needed | | --- | --- | --- | | Correctness | Which operations must agree on current state? | Invariant tests and conflict behavior | | User experience | What delay or pending state is acceptable? | Product acceptance criteria and observed user impact | | Capacity | What workload and failure condition must be sustained? | Arrival distribution, service demand and representative tests | | Recovery | What data-loss and restoration limits apply? | Authorized requirements and restore exercises | | Security | Which identities may access which records and actions? | Threat model and negative authorization tests | | Cost | What recurring and transition costs are acceptable? | Usage assumptions, pricing inputs and staffing needs | | Ownership | Who can deploy, debug and repair the boundary? | Named operating responsibilities and support coverage |

Mark estimates and unknowns. A projected request rate is not a measured workload. A recovery objective is not a demonstrated capability. An architecture can be an appropriate hypothesis while still requiring evidence before production approval.

Constraints can conflict. A stricter consistency requirement may increase coordination or reduce availability during a partition. A cheaper deployment may provide fewer independent failure domains. Record the chosen compromise and the condition that would cause the decision to be revisited.

3. Begin with the smallest credible boundary

Consider a modular application with one transactional database before introducing independent services. It may provide a straightforward place to enforce the reservation invariant. It still requires reliable deployment, authorization, recovery and external-dependency controls.

Split a service when a specific boundary justifies separate ownership, deployment, scaling or isolation. Record the cost of network failure, distributed state, version compatibility, telemetry and on-call knowledge. A repository split alone does not establish operational independence.

Team size is context, not a numerical trigger. A large team can maintain well-defined modules; a small team can struggle with too many services. Inspect coupling and change coordination instead of prescribing a service count from headcount.

The architecture decision should include a retain-current option. If the observed problem is a slow query or a fragile release process, introducing a message bus may not address it. An option is credible only when it changes the constraint responsible for the problem.

Compare the options against the reservation workload, using the same assumptions for each:

| Option | Constraint it can address | New responsibility | Reason to reject it | | --- | --- | --- | --- | | Keep one application and transactional store | Coordinating the reservation invariant in one authority | Module boundaries, query contention, safe deployment | Measured contention or mandatory isolation cannot be solved within this boundary | | Add a derived browse cache | Repeated display reads with permitted staleness | Versioning, invalidation, cache outage behavior | Display correctness requires the latest authoritative state | | Move slow independent work behind a queue | User requests waiting for work that can complete later | Durable acceptance, retry identity, backlog and expiry policy | The product cannot offer a pending result or meet its deadline | | Extract a separately owned service | A demonstrated deployment, isolation, or capacity constraint | Network contracts, independent operation, data ownership | The same database writes and release coordination remain shared |

These are not maturity levels. A team may select the first option for reservations and the third for receipt generation without planning an eventual migration to every other option. The architect should state the observed bottleneck and the evidence expected to change if the option works. If the evidence does not change, stop expanding the design and revisit the diagnosis.

Keep operating capability in the comparison. An option that relies on an unstaffed reconciliation queue or a recovery procedure nobody can execute is incomplete. Budget the tools and operator time needed to own the additional failure mode, including incident review and removal of obsolete components.

4. Separate authoritative operations from derived views

5. Work through a cache race

Writing to a database and then synchronously writing a cache does not, by itself, establish strong consistency between them. Either operation can fail, readers can interleave, and a delayed fill can overwrite a newer value.

Consider this sequence: a reader fetches version 1 from the database but pauses before filling the cache. A writer commits version 2 and invalidates the cache. The reader resumes and installs version 1. Deleting the old cache entry did not prevent stale repopulation.

The following Node.js fixture reproduces that sequence and models a version fence. It intentionally exposes the assumptions rather than presenting a production cache library.

~~~js

// A fence must survive entry eviction and be checked atomically. let minimumVersion = 2; function fill(candidate) { if (candidate.version < minimumVersion) return false; if (cache && candidate.version < cache.version) return false; cache = { ...candidate }; return true; } cache = null; assert.equal(fill(delayedRead), false); assert.equal(fill(database), true); assert.equal(cache.value, "new"); minimumVersion = 3; cache = null; assert.equal(fill(database), false); ~~~

This model prevents the demonstrated stale fill only while the fence is available and correctly advanced. If fence delivery lags behind a commit, a stale read remains possible. Concurrent cache operations need an atomic comparison mechanism; cache eviction must not erase the relevant version protection. The fixture supplies no distributed durability or linearizability proof.

For a correctness-critical read, a simpler decision may be to bypass the derived cache and consult the authoritative system. For browsing, an explicit staleness policy may be acceptable. Test cache outage behavior too: unrestricted fallback can overload the database and turn an optional optimization into an availability dependency.

6. Describe database semantics precisely

Avoid an SQL-versus-NoSQL table that treats one category as always transactional and the other as inherently eventually consistent. Products and configurations offer different transaction scopes, isolation levels, acknowledgement rules and failure behavior. MongoDB, for example, documents multi-document transactions; that does not make its operating constraints identical to another database. MongoDB transactions.

Replication terminology also requires a configuration. PostgreSQL synchronous replication can use selected standbys or a quorum and different acknowledgement stages. “Synchronous” does not universally mean every replica has replayed a write before the client receives success. PostgreSQL 17 standby and synchronous replication guidance.

Write down the consistency required by each operation, the actual isolation and acknowledgement settings, and what happens when the required participants are unavailable. Test failover with those settings. A replica topology diagram without the acknowledgement contract leaves the most important data-loss and availability questions unanswered.

Consider cross-record invariants and external effects separately. A database transaction may protect a reservation row but cannot automatically reverse a dispatched email, an external payment or a warehouse action. Those effects need their own identity, recovery and compensation contracts.

For the inventory example, the database owner compares a conditional atomic update, explicit locking, and a suitable isolation level against the actual invariant. A separate read of available quantity followed by an unconditional decrement leaves a race. The test should coordinate two contenders for the last available unit and verify both the returned outcomes and persisted state.

PostgreSQL 17 transaction-isolation documentation describes the anomalies permitted by each supported level and the need to retry certain failed transactions. An application using serializable transactions must handle aborts and rerun the complete transaction where required. Retrying only the last statement can reuse decisions made from an invalid view of the data. Keep external side effects outside an automatically retried transaction unless their own identity and reconciliation contract makes repetition safe.

The acceptance record identifies the database version, isolation and timeout settings, schema constraints, and test concurrency. A passing single-threaded test cannot substantiate a concurrency guarantee. Repeat the test after changing the ORM, connection pool, retry wrapper, or schema because those changes can alter the operation's effective boundary without changing the architecture drawing.

7. Scope message-delivery claims

At-least-once delivery means repetition must be tolerated under the selected delivery contract. It is not a universal promise that no message can ever disappear. Retention expiry, misrouting, storage exhaustion, acknowledgement timing and unrecoverable infrastructure failures remain relevant.

Distinguish producer acceptance, broker responsibility, consumer processing and business completion. RabbitMQ explicitly separates publisher confirmations from consumer acknowledgements. RabbitMQ confirmations and acknowledgements. An application that acknowledges before its business state is durable can lose work despite receiving the message.

For an event-derived view, define replay provenance and the authoritative history. A log retained for a limited interval cannot rebuild an arbitrarily old projection unless another source exists. Include schema evolution, deletion obligations and access control in the replay design.

Not every application needs event sourcing or separate read and write stores. Adopt those patterns only when their audit, reconstruction, scaling or ownership benefits justify the extra compatibility and operating work. A message bus can be useful without becoming the system's historical source of truth.

If a committed reservation must cause a later notification, identify the gap between committing state and sending the message. One design records an outgoing intent in the same local transaction as the reservation. A relay then publishes that intent and records progress. This moves the unresolved problem into an observable relay lifecycle; it does not make the database and broker one atomic system.

The relay can publish successfully and stop before recording success, so the consumer still needs duplicate handling. The notification owner tracks which business effect was completed and how failed intents are investigated. Define how poison records are quarantined, who can replay them, and how a replay avoids repeating already-completed external work. A dead-letter destination without an owner only stores unfinished obligations.

For every asynchronous boundary, record the oldest uncompleted operation and the evidence that closes it. Queue depth alone cannot tell an operator whether one important reservation notification is permanently stuck behind otherwise healthy throughput.

8. Make capacity a measured argument

Horizontal scaling can increase useful capacity when work can be partitioned and shared dependencies permit it. It can also move contention to a database, connection pool, queue or external API. Adding instances is not a guarantee of a disruption-free deployment or linear throughput growth.

Use a worksheet rather than a fixed headroom multiplier:

| Input | How to obtain it | Why it changes the decision | | --- | --- | --- | | Arrival rate and burst shape | Production observations or an explicitly projected scenario | Average load can conceal short overload periods | | Service demand by dependency | Representative traces and controlled profiling | Identifies the constrained resource | | Effective concurrency | Load tests with real limits and connection behavior | Nominal instance count may not reflect useful parallel work | | Failure condition | Approved availability and recovery requirements | Losing a zone or dependency changes capacity | | Recovery traffic | Replay, backfill, cache warm-up and operator activity | Recovery can compete with normal requests | | Stop condition | Error, latency and resource bounds agreed in advance | Prevents testing beyond authorized impact |

As a worked arithmetic example, assume an isolated worker uses 0.05 seconds of one exclusive execution slot per job. A slot's idealized ceiling is 20 jobs per second before other constraints. This is an invented planning input, not a benchmark. Real arrival variance, contention, retries and downstream waits require measurement.

Do not size production from that arithmetic alone. Use it to identify assumptions for a representative test and explain why the measured result differs. Include degradation behavior when demand exceeds tested capacity.

Capacity protection belongs at the admission boundary. The service owner defines limits on concurrent reservation work, dependency calls, and deferred jobs. When the database is slow, increasing request timeouts can increase the number of occupied connections and make recovery harder. Measure queueing time separately from execution time so a growing wait is not mistaken for a slower business operation.

Test overload together with recovery traffic. A cache restart can send browse misses to the database while reservations and projection rebuilds are also competing for capacity. Reserve or prioritize the correctness-critical path according to the product policy. Degrade optional display detail, defer eligible work, or reject new requests with a clear result before buffers grow beyond their approved bounds. An acceptance test must show both behavior during pressure and how queued obligations finish after pressure stops.

9. Treat serverless as an operating choice, not unlimited capacity

Managed execution changes responsibilities but does not remove quotas, dependencies or cost. AWS Lambda documents concurrency and other limits. Provisioned concurrency also has an explicit capacity and billing model. These facts contradict a blanket claim of infinite scaling or no idle cost for every serverless configuration. Lambda quotas and provisioned concurrency.

For a candidate workload, examine startup sensitivity, execution duration, connection management, retry sources, event age and downstream rate limits. Test the full operation rather than only the function body. A function that scales faster than its database can accept connections may amplify failure.

Compare alternatives using the same workload assumptions. Include observability, network transfer, persistent data, standby capacity, support effort and migration work. A low compute bill can coexist with high operating complexity. A more expensive unit can be worthwhile if it satisfies a constraint the cheaper option cannot.

Build the cost comparison around a completed business operation, not a single invocation. For reservations, record accepted attempts, rejected attempts, retries, completed reservations, retained result records, and notification work. A cheaper handler can still make the operation more expensive if it causes more database reads or repeated downstream calls. Separate these observations from estimates and identify the period and workload represented by each measurement.

The finance and service owners then compare a steady-state scenario with a recovery scenario. The latter includes temporary coexistence, projection rebuilds, restore capacity, investigation telemetry, and retained migration evidence. Record the assumptions that would change the preferred option, such as a longer retention obligation or a lower tolerance for deferred work. Do not hide temporary transition expenditure inside an unexplained annual average.

Cost controls also need a failure policy. A budget alert is an investigation signal, not authority to stop a correctness-critical workflow. Before setting an automatic limit, identify whether it can reject new work safely, strand accepted reservations, or disable the telemetry needed to recover. Name the owner who can approve degradation and the evidence required to remove the limit. This keeps a cost intervention from silently changing the operation contract.

10. Design security and observability across boundaries

Map identities and authorization at each trust boundary. Encryption in transit is not authorization. A valid service identity should not automatically allow access to every tenant or administrative action. Test denied paths and least-privilege behavior, including recovery tools.

Record sensitive-data flows, retention and administrative access. Logs and traces can accidentally create a second copy of personal or secret data. Choose fields deliberately and apply access and deletion controls to telemetry.

Debugging distributed work requires correlation, useful state transitions and ownership. A service mesh can provide some infrastructure telemetry, but debugging is not impossible without one, and a mesh cannot explain every business-state transition. Combine application events, metrics and traces according to the questions operators need to answer.

Define the symptoms that should page an operator, the context available at that moment and the action they can safely take. A dashboard is not a recovery procedure. Exercise access loss, dependency timeout and partial deployment so the operating model is tested alongside the software.

Security boundaries also change the operational design. A reservation worker may need access to inventory but not customer payment credentials. A replay tool may need a narrowly scoped capability to retry one operation, not permission to impersonate every user. Record the identity, permitted action, data scope, and audit destination for each ordinary and emergency path.

NIST SP 800-207 frames zero trust around explicit authentication and authorization rather than implicit trust from network location. Apply that principle to the operation: an internal call or restored database does not by itself establish the caller's authority. This is a design input, not a claim that the reference architecture implements the entire NIST model.

The security owner tests a revoked tenant member, a worker with excessive privileges, and an operator attempting an out-of-scope replay. The operations owner tests how to recover when the identity dependency is unavailable without introducing an undocumented bypass. Emergency access should have an approving authority, bounded scope, expiration, and a reviewable record. Include those responsibilities in the service's recovery objective; a technically restorable system can remain unusable if nobody can obtain authorized access.

11. Compare failure modes and transition costs

For each option, ask what fails first, what remains available, and how the team knows the difference. A timeout, stale read, duplicate command and unavailable dependency can require different user responses.

Test at least one partial failure, not only complete outage. A dependency that sometimes responds slowly can exhaust resources differently from one that fails immediately. Introduce faults only in an authorized environment with bounded impact.

Compare migration states as well as end states. Running old and new systems together creates synchronization and ownership costs. Routing back does not undo data changes or external actions. Define compatibility and the point after which forward repair or data recovery replaces simple traffic reversal.

Retain an explicit “do not proceed” condition. If a design requires operating skills, budget or recovery evidence that the organization cannot provide, the architecture is not ready merely because its components are available to purchase.

12. Reusable architecture decision record

Use this template for a consequential boundary:

Decision and owner: the specific choice, accountable role and review date.
Operation contract: acceptance, completion, invariants and permitted degradation.
Context: workload, security, recovery, cost and organizational constraints.
Options: retain current design, smallest credible change and more extensive alternative.
Chosen boundary: authority, transaction scope and derived state.
Failure behavior: timeouts, duplicate work, stale data and operator intervention.
Evidence: observed tests, source versions, assumptions and unanswered questions.
Transition: compatibility, staged rollout, recovery and irreversible steps.
Revisit trigger: a measured condition or changed requirement that invalidates the choice.

Keep the record short enough to update, but link to reproducible evidence. A decision that cannot identify its assumptions is difficult to revisit fairly. A decision that records a test failure and a revised boundary can be stronger than a polished diagram with no adverse evidence.

13. Resolve an uncertain commit before repeating work

A timeout tells the client that it did not receive a result in time. It does not establish whether the database committed. The following sequence shows a single logical reservation whose result is retrieved after a retry. It omits payment and inventory expiry so the commit boundary remains visible.

The application owner defines how an uncertain operation is queried and how long its result remains available. A retry can re-run authorization without creating another business effect. If the stored operation parameters differ, reject the reuse and retain evidence of the conflict. If the result record is unavailable, return a bounded pending or recovery state rather than claiming either success or failure without evidence.

Test the response-loss window with a controlled fault after commit and before response delivery. Then issue simultaneous retries with the same identity. Verify one reservation, one stored result, correct access control, and a comprehensible client outcome. Repeat with a handler restart and a delayed database response. This evidence is more useful than claiming that an endpoint is idempotent without defining its identity and retention boundary.

14. Design the transition as a separate architecture

An eventual target diagram omits the riskiest period if old and new implementations coexist. Draw the writer authority, readers, and synchronization mechanism for each transition stage. The migration owner records which version can read and write each representation, how progress is measured, and what must be reconciled before the next stage begins.

For a reservation-schema change, add compatible fields before requiring them. Deploy readers that understand both representations, backfill under resource limits, and compare invariants before switching writers. These are proposed stages, not a universal migration recipe. A required type conversion, deletion, or external contract may change the available order and rollback options.

Separate traffic rollback from data recovery. Before the new writer creates an incompatible value, returning traffic to the old version may be sufficient. Afterward, the old version may reject or misinterpret valid new records. The release owner must know the last reversible stage and the repair-forward procedure before crossing it. A feature flag cannot reconstruct an overwritten value or cancel an external warehouse action.

Use a cutover ledger with stage, schema revision, writer identity, synchronization checkpoint, rejected records, reconciliation result, and approving owner. Stop if source and target disagree on reservation quantity or ownership. Preserve both observations and investigate; do not change the comparison until the unexplained difference disappears. Acceptance requires a useful business operation after recovery, not only a green deployment status.

15. Make the evidence pack challenge the design

The architect proposes tests that could disprove the selected boundary. For the inventory scenario, the minimum useful pack combines concurrent last-unit reservations, response loss, stale cache repopulation, queue replay, denied tenant access, and recovery from a known checkpoint. Select workload and failure conditions from the actual product requirements; this list is a starting point, not a security or reliability certification.

| Review question | Acceptance evidence | Owner who can reject release | | --- | --- | --- | | Does the invariant survive contention? | Coordinated concurrent test and persisted-state reconciliation | Data owner | | Can users recover an uncertain result? | Response-loss exercise and stable operation identity | Application owner | | Can an optional component fail safely? | Cache or notification outage with bounded resource demand | Service owner | | Are recovery actions authorized? | Denied-access tests and recorded emergency access | Security owner | | Can another operator restore service? | Independent runbook exercise and business checks | Operations owner |

Record the tested revision and all material settings. Keep failures and unresolved gaps beside successful results so the review does not reward a carefully selected happy path. If a test needs production data, workload, or privileges, obtain the required authority and handling controls first. An evidence gap should change the release decision or create a bounded condition, not become an invented pass.

Limitations and next step

Primary references were checked on 21 September 2026. The cache fixture tests a local ordering model, not production consistency. The capacity example uses invented assumptions. No named-company architecture, anonymous case result or quantified improvement is offered as proof.

An accountable author, architecture reviewer and deployment-specific validation are still required before factual and publication approval. The document has been editorially corrected, but that does not authorize a customer claim or approve an implementation. No unverified downloadable PDF is attached.

For a scoped system architecture review, bring one critical operation, current dependency map, failure evidence and constraints. A useful first result is an explicit decision record and test plan that can disprove the proposed design as well as support it.