SageMaker Endpoint Updates: Prove the Rollback Signal Before Expanding Traffic

Map a SageMaker endpoint update to the right fleet, alarm and observation window. Use negative packets to distinguish serving faults, missing evidence and delayed...

Primary sources checked 2026-10-09

A healthy endpoint is not necessarily a healthy candidate

Before updating a SageMaker endpoint, identify which fleet supplies each stop signal. An alarm across the entire endpoint can conceal a failing candidate behind healthy incumbent traffic. A candidate-specific metric can avoid that mixture, but an empty candidate stream still supplies no evidence. Establish eligibility, metric identity, observation coverage, alarm-read permission and a usable recovery package before the update. Keep prediction-quality acceptance separate from serving-health observations.

This worked scenario produces a rollout-observation packet for one fictional support-ticket classifier. Its incumbent configuration is support-v7; the proposed configuration is support-v8. The endpoint uses one provisioned real-time variant named AllTraffic, with a blue/green canary update. This is predictive model serving, not an LLM-provider migration, multi-model routing or training workflow.

The examples and supplied local evaluator are synthetic. No endpoint, alarm, permission or AWS account was inspected. No traffic was sent and no update or rollback was executed. The evaluator checks a limited evidence contract and arithmetic, not CloudWatch alarm behavior. The figures distinguish the lifecycle, measured populations and decision owners.

The production MLOps article owns package compatibility and delayed quality evidence. The deployment acceptance pack owns the broader release manifest and recovery rehearsal. Use this packet inside those controls when the exact SageMaker fleet-to-observer relationship is the unresolved decision.

Choose a supported change before choosing an alarm

AWS describes deployment guardrails for real-time and asynchronous inference, with distinct blue/green and rolling arrangements. This exercise admits only the provisioned real-time, single-variant blue/green canary scope above. Being outside that teaching scope is not a claim that AWS offers no other supported arrangement. Guardrail scope.

Check the proposed endpoint configuration, not just today's working configuration. AWS requires matching variant names and lists Marketplace containers and Inf1 endpoints as guardrail exclusions, with an all-at-once fallback lacking the final baking period. Rolling has additional exclusions; do not transfer those rules indiscriminately to blue/green. An excluded endpoint is not a canary merely because the request contains a canary-looking setting. Current exclusions.

Compare three alternatives. A supported guarded update keeps the service's deployment mechanism and its reviewed rollback alarms. A separately provisioned replacement endpoint gives the application routing owner a different exposure control, but requires its own permissions, capacity, routing, observation and reversal design. Retaining the incumbent is appropriate when eligibility or evidence is unknown. It is a decision with a named next check, not an indefinite “monitor closely” instruction.

Do not widen this packet to serverless, asynchronous inference, multiple variants or rolling by changing a title. Collect their actual configuration and research that mode before reuse. The local evaluator deliberately returns HOLD for other scopes. Its feature checks are a small selection boundary, not an exhaustive eligibility validator.

Freeze the update identity and effective configuration

Record endpoint name, account/Region evidence location, old and proposed endpoint configurations, variant, model/package references, image and preprocessing compatibility, and the owner accepting the change. Use aliases in a shared packet; keep restricted identifiers in an approved evidence store. A model package version alone does not identify the endpoint configuration or its effective traffic properties.

UpdateEndpoint takes a new endpoint configuration. Its deployment settings can be supplied or explicitly retained, and variant-property retention affects effective instance counts and weights. Keep the chosen retention settings with the request. A returned endpoint ARN acknowledges the request; subsequent status and configuration readbacks are separate evidence. Do not delete an endpoint configuration in use or during an update. UpdateEndpoint contract.

Write down what the resulting effective values should be, rather than assuming the new configuration file is the entire runtime state. A reviewer should be able to compare a request revision and an observed endpoint revision without reconstructing choices from memory. If an operator changes instance counts, alarm definitions or candidate package after approval, invalidate the affected packet and re-evaluate it.

Preserve the prior compatible package and its dependencies before starting. Recovery requires more than the old configuration's name: its artifacts must remain available, its runtime must still work, and downstream schema or application changes must not make the old predictions unusable. This guide does not establish those conditions from a supplied reference string.

Map traffic, telemetry and control as different paths

Blue/green replacement creates a second fleet, shifts serving traffic and later retires the old fleet. During overlap, both fleets can incur cost. An alarm during the reviewed baking phase can trigger service rollback. The old fleet's continued presence and the monitoring interval matter; “we can always roll back” is not an adequate recovery statement. Blue/green lifecycle.

For canary mode, AWS limits canary size to half the new fleet's capacity. Capacity percentage is not a measured percentage of business requests. Record canary size, wait interval, termination wait and overall execution timeout separately. The observed candidate request count, client concurrency and request mix determine what the canary actually exercised. Canary controls.

Before approving exposure, ask whether the planned observation can finish while the relevant deployment safeguard remains active. Include time for candidate traffic, metric arrival, alarm evaluation, operator notification and the authorized response. If no one can demonstrate that relationship for the selected settings, hold before initiation. This guide supplies no universal timing formula and no SageMaker pause command. During an active update, use the already approved service/operator response; do not assume the local HOLD result suspends traffic shifts.

Estimate overlap capacity and its financial limit from the selected deployment plan and current account evidence. Check quotas and availability separately. No instance price, quota entitlement or deployment duration is assumed here. A cost cap may be an operator stop criterion, but it does not replace serving or prediction-quality criteria.

*Fictional provisioned real-time, single-variant blue/green canary. Gray arrows show phase order, not duration or measured request share. Blue arrows carry requests, purple dashed arrows carry telemetry and red arrows carry rollback control. Desktop draws both telemetry branches; mobile names each population's inputs instead. Return to the incumbent is conditional on the reviewed safeguard and retained fleet. HOLD is an owner decision, not an AWS pause. No update, alarm transition or rollback was executed.*

Phase order is conceptual, not a duration or request-share claim. The retained-fleet safeguard is bounded; returning after retirement requires a new authorized change. No deployment or rollback was executed.

Blue arrows carry requests; purple dashed arrows carry telemetry. Only the candidate supplies its specific population. Latency and the separate error example use stipulated counts, not observed AWS telemetry or CloudWatch alarm emulation.

Purple dashed arrows carry telemetry; red carries alarm-driven control. Return remains conditional on retained v7 and the reviewed lifecycle. Missing evidence is an owner HOLD, not a service pause. Serving evidence does not approve prediction quality.

Pin the signal population, units and observation window

AWS's rollback configuration guidance distinguishes combined old/new fleet monitoring from new-configuration ModelLatency, whose documented dimensions include EndpointName, VariantName and EndpointConfigName. It also requires the SageMaker execution role to read configured alarms through cloudwatch:DescribeAlarms. A combined-fleet fault can originate in the incumbent. Candidate metrics begin with candidate traffic, not merely creation of a configuration. Alarm and population configuration.

That documented new-configuration latency example does not prove that every endpoint metric supports every added dimension. Preserve the exact metric definition and retrieved stream. If a query with a plausible dimension returns nothing, do not silently fall back to an endpoint aggregate and call it equivalent. Record the changed population and reopen the decision.

ModelLatency is measured in microseconds and includes model-container processing and local communication. It is not the client's complete response time. AWS documents request-count and error metrics separately; distinguish a count, a per-request indicator, an average and a percentile before comparing them. Endpoint metric definitions.

The filled example uses a local counted-window mean. It stipulates two complete consecutive 60-second windows, at least 20 candidate observations in each, and a stop review when both means exceed 200,000 microseconds. Those are fictional owner-selected teaching criteria, not AWS defaults or recommended production thresholds. Equality does not breach this local criterion. One breaching window remains unresolved instead of being averaged away.

A mean can hide a slow tail or a failure concentrated in one input slice. A real acceptance owner must select the signals appropriate to the classifier's contract, including client-observed timeouts, malformed outputs and important slices. Do not infer p95 latency or rare-fault detection from 20 requests. Record any untested slice and its exposure limit explicitly.

Treat missing data and missing permission as unresolved evidence

CloudWatch offers missing, ignore, breaching and notBreaching policies. Its evaluation can retrieve additional datapoints and use existing observations before applying a missing-data policy. Consequently, two local windows are not a CloudWatch alarm emulator. A recorded OK state with notBreaching does not establish that the candidate received representative traffic. Missing-data evaluation.

Keep two questions separate: what the configured alarm is expected to do, and whether enough candidate evidence exists for the owner's release decision. The local fixture holds absent or undersampled windows under every recorded missing-data policy. Real alarm state, state reason and timestamps must be read separately. Do not claim that the fixture predicts when an actual alarm changes state.

Permission evidence must identify the relevant execution role, alarm references, scope, collector and collection revision. An engineer's ability to open an alarm does not establish the deployment role's ability to read it. Conversely, an access denial while collecting evidence does not prove the alarm is absent. Mark permission unknown and obtain an authorized check; do not broaden access as a shortcut.

Without configured alarms, the deployment does not have alarm-driven automatic rollback. The packet needs both the alarm definition and proof that the selected service-side rollback configuration references it. A screenshot of a healthy chart establishes neither. Negative testing in a separately authorized environment should include the failure the alarm is supposed to detect, not simply a successful request. Preserve request identity, actual candidate observations, alarm transitions and resulting endpoint configuration.

Work two misleading populations before trusting the happy path

In the first synthetic window, the incumbent supplies 180 observations totaling 18,000,000 microseconds. Its mean is 100,000 microseconds, or 100 ms. The candidate supplies 20 observations totaling 8,000,000 microseconds, with a mean of 400,000 microseconds, or 400 ms. The mixed mean is 26,000,000 divided by 200: 130,000 microseconds, or 130 ms.

Repeat those stipulated counts in the second window. The 200 ms local stop threshold is exceeded by the candidate in both windows, but not by the mixed population. The evaluator reports STOP_REVIEW for candidate observations. It does not report that AWS rolled back or that the combined population is an acceptable substitute. This is counted arithmetic, not a simulation of CloudWatch ingestion or aggregation.

The second example reverses the attribution problem. Among 180 incumbent requests, 18 fail; among 20 candidate requests, none fail. The incumbent rate is 10%, candidate rate 0%, and combined rate 9%. A stipulated 5% aggregate error criterion would be exceeded, but blaming the candidate would be unsupported. Request-count arithmetic here does not implement the service's error metric statistics. Preserve the aggregate incident while investigating which fleet and failure category contributed.

The favorable example has candidate means of 100 ms in both local windows, but prediction-quality labels have not matured. Its result is HOLD_QUALITY, not accepted. A support classifier can respond quickly while routing tickets incorrectly. When labels arrive, the quality owner must review the agreed slices and acceptance rule; a packet value stating “reviewed” is not independent proof of that review.

Complete a readable rollout-observation record

Use the following filled worksheet as a model for structure, not a copyable production configuration. Every reference beginning with synthetic/ is an invented teaching reference. The fixture checks declared identities and numbers; it cannot authenticate those references.

Filled fictional record

Packet, owner and evidence

fictional-r1; fictional ML release owner; synthetic/update-record. Account and Region are not supplied or inspected. Relative intervals below have no real UTC counterpart.

Endpoint and configurations

support-classifier-demo; incumbent support-v7; candidate support-v8; unchanged variant AllTraffic. Prior compatible package recorded in synthetic/prior-package-record.

Mode and effective properties

Provisioned real-time, single-variant blue/green canary. Proposed 20% new-fleet capacity, not a guaranteed request share. Deployment and variant-property retention choices must be explicit in any real request; they are not executed by this fixture.

Eligibility and capacity

Marketplace false and Inf1 false are stipulated. synthetic/capacity-record represents a fictional reviewed capacity assumption. Real exclusions, quota, availability, effective counts and overlap budget remain separate checks.

Observer identity

Namespace AWS/SageMaker; ModelLatency; EndpointName support-classifier-demo; VariantName AllTraffic; EndpointConfigName support-v8; unit Microseconds. synthetic/latency-alarm-description is not an actual alarm ARN.

Criterion and coverage

Local sum divided by count; two contiguous windows [0,60) and [60,120); at least 20 candidate observations each. Both means strictly above 200,000 microseconds trigger STOP_REVIEW. One breach produces HOLD.

Missing-data and permission evidence

Recorded policy missing; absent candidate observations always hold the local decision. synthetic/permission-record stipulates alarm-read evidence, without testing a role or policy.

Serving result

Favorable fixture: 20 observations and 2,000,000 microseconds in each window, yielding 100,000 microseconds per mean. The bad-candidate mutation instead supplies 8,000,000 per window and yields 400,000.

Quality and recovery

Labels unavailable, so serving-only outcome HOLD_QUALITY. Prior compatibility is stipulated, not restored. Real recovery owner must establish package availability and downstream compatibility before exposure.

Decision and next action

No AWS change is authorized. Review missing quality evidence and the actual endpoint/alarm packet. Before any separately authorized rehearsal, replace synthetic references, select actual timing and stop controls, and identify who reads recovery results.

Blank reusable record

Copy these labels into an approved evidence location. An empty field means unknown, not “not applicable.” Record an explicit reason and owner for exclusions.

Packet, owner and evidence

Packet revision: ____; accountable release owner: ____; collector and UTC collection time: ____; restricted evidence location: ____.

Endpoint and configurations

Account/Region: ____; endpoint: ____; old/new configuration: ____; variant: ____; package/image/preprocessor revisions: ____.

Mode and effective properties

Exact mode: ____; canary capacity choice: ____; retained deployment/variant settings: ____; expected effective counts/weights: ____; actual readback: ____.

Eligibility and capacity

Feature inventory and exclusion review: ____; quota/availability evidence: ____; overlap capacity/budget: ____; owner for unresolved scope: ____.

Observer identity

Alarm reference and deployment association: ____; namespace/metric/dimensions: ____; population: ____; unit/statistic: ____; definition revision: ____.

Criterion and coverage

Owner threshold and comparison: ____; period/window/alignment: ____; required observations/slices: ____; emission/evaluation allowance: ____; exposure limit: ____.

Missing-data and permission evidence

Missing-data choice and rationale: ____; actual alarm state/reason/time: ____; execution-role alarm-read evidence: ____; denied/unknown evidence owner: ____.

Serving result

Candidate-window identities and observations: ____; client faults: ____; count/statistic reconciliation: ____; bounded conclusion: ____; untested slices: ____.

Quality and recovery

Quality labels/window/slices/owner gate: ____; prior compatible package and retained dependencies: ____; stop/recovery authority: ____; recovery readback criteria: ____.

Decision and next action

Eligibility/evidence/serving/quality decisions: ____; authorized bounded next action: ____; owner/deadline: ____; packet-invalidating changes: ____.

These records use one text flow on desktop and mobile rather than a wide matrix. Keep complete labels and values when adapting them; a mobile summary that omits configuration identity would remove the information needed to detect the wrong stream.

Run the offline counterexamples without implying an AWS rehearsal

Download the four companion files into one folder: declared packets, evaluator, executable tests and README and limitations. Use Node.js 22 or later and run node --test observation.test.mjs from that folder. The 23 behavior tests use built-in modules and make no network calls. Expected outcomes are declared separately from the classifier implementation, with direct arithmetic assertions for both mixed-population examples. Repository-only manuscript checks are not part of this standalone download.

Its negative packets include wrong candidate configuration, wrong endpoint, missing candidate data, unknown alarm-read permission, excluded Marketplace features, unknown feature inventory, missing recovery evidence and unreviewed capacity. Additional tests reject wrong observed identity, stale/duplicate intervals, nonnumeric counts, unsafe aggregate totals and another deployment mode. Changing missing-data policy cannot rescue absent candidate observations.

The evaluator accepts only two contiguous local 60-second windows. It does not model metric delays, probabilistic sampling, CloudWatch M-of-N or percentile alarms, concurrent endpoint updates, actual service permissions, rollback timing or mature prediction quality. REVIEW_READY means the limited packet warrants owner review. Every outcome retains authorizesAwsChange: false. Do not turn this output into a deployment trigger.

Close the serving decision without closing the wrong gate

Before initiation, HOLD means do not start the proposed change until its missing evidence is resolved. During a running update, a stop condition invokes the pre-authorized response, with actual endpoint and alarm readbacks. A failed signal contract may require retaining the incumbent or using a separately controlled replacement design; it is not repaired by accepting an aggregate that happens to look favorable.

After rollback, verify which configuration is serving, its effective variant properties, representative client behavior and retained recovery dependencies. A service status alone does not repair wrongly classified tickets, duplicate downstream actions or application changes made during exposure. Assign those consequences separately. If the old fleet has been retired, returning to the prior package requires a new reviewed change and capacity path, not an assumption of instant reversal.

The completed packet should make four conclusions explicit: whether this configuration/mode is eligible, whether the evidence actually observes the candidate, what serving behavior the bounded windows support, and which prediction-quality gate remains open. Accept only the conclusion supported by its evidence. That is more useful than calling an endpoint “green” when nobody can say what green measured.