Rightsizing Compute Without Breaking Things

Right-size compute from workload evidence, validate runtime and failure capacity, and roll out changes with separate stateless and data-bearing recovery plans.

Change capacity only after identifying the constraint

Low average processor (CPU) utilization is a reason to investigate, not permission to halve an instance. The workload may be constrained by memory, storage latency, network throughput, connection limits or a peak hidden by the average. A smaller configuration may also remove the spare capacity required during a zone failure.

This playbook covers the engineering decision to change compute size, family, architecture or container resources. It does not assume a typical waste percentage or a guaranteed savings rate. Use the compute purchasing playbook for commitments after the workload shape is understood.

Start with one owned service. Record why its current capacity exists. Some headroom may protect a recovery objective or absorb demand while new instances start. Other capacity may be accidental. The change should distinguish the two.

1. Define the outcome and the stop conditions

The service owner selects the user-facing measures that must remain acceptable: successful transactions, tail latency, deadline completion, correctness and durability. Add an expected demand range and the failure conditions the service is designed to survive.

Write the abort conditions before running the candidate. They may include a regression against the service objective, sustained queue growth, memory exhaustion, replica lag, insufficient recovery capacity or a correctness failure. Thresholds must follow the workload's objective and baseline variability, not a generic CPU percentage.

The FinOps owner defines the cost view. A lower instance rate may not lower this month's bill if existing commitments remain unused. Report both resource consumption and realized billed cost, with workload volume and amortization assumptions. Avoid calling an internal allocation change a cash saving.

Gate: the owner agrees what evidence would accept or reject the change. If the team cannot observe the customer outcome or the likely resource constraint, improve that instrumentation first.

2. Collect evidence across the workload cycle

Include peak demand, quiet periods, deployments, batch windows, cache warm-up and known recovery events. Choose an observation period that captures those conditions. If an annual event is missing, use an explicit forecast or controlled test and record the uncertainty.

Confirm that the monitoring system collects the needed memory and disk signals. Provider-level instance metrics may not include the process or operating-system measurements required for the decision. Missing data is not evidence of unused capacity.

Vendor recommendations can generate candidates, but inspect their assumptions, supported metrics and observation window. Verify current feature availability and charges before enabling additional collection. Do not label every recommendation service or telemetry feature free.

3. Form one testable sizing hypothesis

The platform engineer writes the proposed change and expected mechanism. “Reduce API worker memory because the measured working set fits with approved headroom” is testable. “Use the cheaper family” is incomplete.

Compare candidate constraints using the provider's current specifications. The EC2 instance-type guide distinguishes compute, memory, storage and networking capabilities. Similar vCPU counts do not guarantee equivalent application throughput.

For AWS examples, M6g uses Graviton2 and M7g uses Graviton3. Verify the exact family rather than inferring processor generation from a general “Graviton” label. The M6g and M7g product specifications support that distinction. They do not establish which instance is cheapest for your workload.

Change one major constraint at a time where practical. Combining architecture migration, replica reduction and a database upgrade makes a regression harder to attribute and recovery harder to validate.

4. Test representative demand and failure capacity

Build a reproducible test with the same application version, relevant data distribution, request mix and dependency behavior. Control external side effects with a sandbox or a verified no-side-effect path. Do not replay production traffic into payment, email or customer-write endpoints without that protection.

Run baseline and candidate under comparable conditions. Record warm and cold behavior, concurrency, tail latency, errors, queue age, resource saturation and cost per accepted unit of work. Explain sampling limits and any test environment differences.

Then test the failure condition the service is meant to tolerate. Remove an instance or simulate the loss of an intended failure domain within an approved environment. Verify that remaining capacity and scaling can sustain the agreed degraded objective. Headroom should follow this requirement, not a universal multiplier.

Gate: the candidate meets the approved objectives under the tested workload and failure conditions. If the limiting resource moved from CPU to memory or I/O, revise the hypothesis instead of treating the smaller instance as a completed optimization.

5. Coordinate Kubernetes requests, limits and autoscaling

Container requests influence scheduling and some autoscaling calculations. Limits constrain runtime behavior. Reducing requests may improve packing, but it does not remove nodes or reduce the cloud bill by itself.

If Vertical Pod Autoscaler is installed, recommendation-only mode can provide evidence without automatically changing pod resources. The upstream VPA quick start documents updateMode: "Off". Verify the installed version, CRDs, target workload and recommendation coverage before adopting its output.

Review the interaction with Horizontal Pod Autoscaler. For utilization-based resource targets, HPA calculates utilization relative to resource requests. Changing the requests can therefore change scaling behavior even when absolute resource use is unchanged. See the Kubernetes HPA documentation.

Use a workload-specific sequence:

1. Inspect recommendations and observed resource distributions without enabling automatic changes.
2. Review requests, limits, HPA targets and replica bounds together.
3. Test startup, throttling, memory pressure and scaling on the candidate.
4. Confirm node capacity, scheduling constraints and failure-domain placement.
5. Release a bounded workload cohort and observe both pod and node behavior.
6. Consolidate nodes only after disruption, rescheduling and recovery checks pass.

A PodDisruptionBudget helps constrain voluntary disruptions; it does not guarantee capacity during an involuntary outage. Keep application availability, graceful termination and spare placement capacity in the acceptance test. Do not enable multiple controllers to adjust the same resources without understanding their feedback loops.

6. Treat database rightsizing as a data-bearing change

The database operator checks working-set memory, cache behavior, query plans, locks, connection limits, I/O, replication lag and recovery time. No single cache-hit percentage or CPU threshold establishes that a database can shrink safely.

Use an isolated, appropriately protected dataset for load tests. Query log analyzers help identify workload patterns; they do not by themselves replay production transactions faithfully. A replay tool must preserve the relevant concurrency and transaction behavior while preventing external effects or accidental production writes.

Before cutover, verify backup restoration and the chosen replication or migration path. Define the authoritative writer and fence competing writers. Retaining the old database for a fixed number of hours is not a safe rollback plan once the new database has accepted writes.

Do not silently discard new writes to recover a performance metric. If recovery objectives cannot be met with the proposed migration method, defer the resize or choose another approach.

7. Release a bounded cohort and keep recovery ready

The service owner chooses cohort scope and observation conditions from traffic and risk. The cohort must see representative work, including relevant peaks and background processing. A quiet period does not demonstrate peak readiness.

For stateless workers, keep a known-compatible configuration available. On an abort signal, stop expansion, route new work to healthy capacity, drain workers where possible and reconcile leases or in-flight requests. Verify quota and actual capacity for the recovery configuration; an old launch template alone cannot create unavailable instances.

For data-bearing systems, use the cutover-state decision above. Application rollback and data rollback are separate operations. Record which changes are reversible, which require reconciliation and who may authorize each.

Keep the previous configuration until the acceptance window and recovery evidence are complete, then remove obsolete capacity through the normal change process. Set an owner and deadline so temporary duplicate capacity does not become permanent spend.

Reusable rightsizing change record

Service / owner / change ID:
Current and candidate configuration:
Sizing hypothesis and expected constraint:
Telemetry source, observation conditions and missing data:
User-facing objectives and abort conditions:
Compatibility, license and architecture checks:
Load-test inputs, baseline and candidate results:
Failure-capacity and autoscaling evidence:
Stateful cutover point and authoritative writer:
Rollback or fix-forward procedure and reconciliation owner:
Dated prices, commitment effects and expected cost range:
Cohort, observation conditions and expansion approver:
Observed outcome, residual risk and cleanup owner:

The record should let another operator reproduce the decision and stop the rollout. Attach raw measurements or dashboard snapshots with dates, not only a conclusion that the candidate “looked healthy.”

Run a change review that separates savings from transferred risk

Before expanding the cohort, bring the service owner, platform owner and finance partner through the same evidence. The service owner validates user-facing behavior and recovery obligations. The platform owner checks scheduling, quotas, autoscaling and shared-capacity effects. The finance partner checks whether the comparison includes commitments, licenses, data transfer and temporary parallel capacity.

Use a simple evidence table rather than a single savings percentage:

A candidate that costs less but removes required headroom is not right-sized. It is under-provisioned. A candidate that preserves service behavior but transfers work to a more expensive database or third-party API may also fail the economic objective. Record both direct and transferred effects.

Exercise the first-hour operator scenario

Ask an operator who did not design the change to respond to a simulated regression. Provide only the normal alert and runbook. The operator should identify the changed capacity, distinguish demand growth from saturation, find the approved stop condition, and execute or explain the recovery path.

For stateless capacity, this may mean restoring the prior configuration and confirming that scheduling and autoscaling converge. For a data-bearing system, the operator must also identify the authoritative writer, writes accepted since cutover, replication state and reconciliation steps. If recovery depends on undocumented knowledge or a person who is unavailable, the rollout is not ready.

Retain the rehearsal output with the change record. The point is not theatrical incident response. It is to prove that the claimed reversibility exists under the same access controls and tooling that will be available during a real event.

Verify the result and record limitations

• The sizing hypothesis names the actual resource constraint
• The candidate passed compatibility, representative-load and failure-capacity tests
• Requests, limits and autoscaling interactions were reviewed together
• Database recovery accounts for writes accepted after cutover
• The cohort remained within approved service objectives
• Cost comparisons use comparable workload volume and commitment accounting
• Temporary capacity and configuration changes have cleanup owners
• Untested conditions and residual risks are recorded with the service owner's decision

Compare accepted work, latency, errors, recovery behavior and realized cost after rollout. If demand fell during the observation period, report that confounder. A lower bill does not isolate the effect of rightsizing.

This method cannot guarantee future peaks or provider capacity. Test data, seasonal demand, external dependencies and contractual pricing all limit the conclusion. Revisit the decision when workload shape, runtime, business objectives or recovery requirements change. Bring the change record to cloud cost optimization if the engineering and financial evidence disagree.

Schedule the next review against a known demand cycle, not an arbitrary calendar reminder.

Primary references

Processor-family and platform behavior references checked on September 21, 2026. Recheck the deployed versions and exact instance specifications before a change.

• AWS: EC2 instance types and hardware specifications
• AWS: M6g instance specifications
• AWS: M7g instance specifications
• Kubernetes: Horizontal Pod Autoscaling
• Kubernetes Autoscaler: VPA quick start
• Kubernetes: Disruptions