Kubernetes Production Readiness: Evidence Before Launch

Review Kubernetes capacity, probes, disruption, recovery and upgrades with workload-specific tests instead of universal resource ratios or configuration counts.

A workload is ready for production when its owners can show how it behaves under expected demand, failures and maintenance. A running Deployment proves that containers started. It does not prove that the service survives a node loss, drains work safely or restores its data.

Assess a named workload and the cluster services it depends on. Record the Kubernetes and provider versions, node operating systems, network and storage plugins, admission policies and enabled features. The examples here concern Linux workloads using container-level resource settings. Managed services and newer resource features can change the applicable controls.

Use this guide for the launch decision. The Kubernetes security architecture guide covers the security design in greater depth.

Write the service acceptance conditions first

State what customers need, including correctness and completion deadlines. A request-serving API, a scheduled report and a queue worker require different evidence.

| Decision | Evidence to bring | | --- | --- | | Availability | Critical journeys, acceptable interruption and dependency behavior | | Capacity | Expected demand, peaks, startup load and recovery traffic | | Correctness | Duplicate handling, partial writes and business reconciliation | | Recovery | Agreed recovery point and recovery time, demonstrated by a restore | | Ownership | On-call contact, change authority and escalation coverage |

Make assumptions visible. If the service depends on a single external database, extra application replicas do not remove that dependency. If a rare batch peak has never been tested, record the gap rather than substitute a dashboard average.

Size requests and limits from workload behavior

Requests influence scheduling; limits constrain runtime resource use. In the conventional Linux model, CPU limits can throttle execution, while memory limits can lead to OOM termination. The timing and behavior are not identical. Consult the current resource-management documentation for the selected node and feature configuration.

Observe startup, steady load, bursts, garbage collection, dependency slowdown and shutdown. A percentile of normal usage may miss the peak that determines whether a pod starts or survives recovery. A limit set at twice the request has no general safety meaning.

For a latency-sensitive service, test the effect of CPU throttling before selecting a CPU limit. If you omit that limit under an approved policy, still bound concurrency and protect the node with appropriate requests, quotas and capacity planning. For memory, include relevant caches, native allocations and memory-backed temporary storage, not only the language heap.

QoS classification follows the configured resource model. With container-level settings, Guaranteed requires positive CPU and memory requests and equal corresponding limits for every container. QoS is not a business-priority label or a guarantee against all failures. Node-pressure eviction also requires understanding priority, resource use and the node's eviction policy.

Worked capacity check: scheduling can pass while memory fails

Assume a fictional worker pool has 1,500 millicores and 1,536 MiB of capacity available to this workload after other commitments. Four replicas each request 300 millicores and 256 MiB. A rolling update allows one additional replica.

| State | Requested resources | | --- | --- | | Four replicas | 1,200 millicores and 1,024 MiB | | Five during rollout | 1,500 millicores and 1,280 MiB |

The five-pod request total fits the stated budget. Now suppose testing shows that each worker can use 450 MiB at the same time during a recovery burst. Five workers then need 5 × 450 = 2,250 MiB, which exceeds the 1,536 MiB budget by 714 MiB.

The example exposes an inadequate planning assumption; it is not a recommended configuration. A higher memory limit alone does not create capacity. Revisit requests, usable node capacity, burst concurrency and rollout settings, then repeat the test.

Also test scheduling by node and failure domain. An aggregate budget can conceal fragmentation or topology constraints. Autoscaling is not immediate capacity: account for detection, node startup, image pulls, initialization and quotas.

Give each probe one operational purpose

The Kubernetes probe documentation describes optional checks with different consequences.

| Probe | Use it when this action is appropriate | | --- | --- | | Startup | Initialization needs time before ongoing health checks should run | | Readiness | The workload should stop receiving ordinary Service traffic until it can serve it | | Liveness | Restarting a stuck container is likely to restore progress |

A startup probe, when configured, prevents liveness and readiness checks from running until it succeeds. A failed readiness check does not itself restart a container. Liveness failure can restart it according to the configured behavior.

Do not use a remote database outage as an automatic reason to restart every otherwise healthy API process. That can replace a dependency outage with a restart storm. Decide whether the service can serve a degraded response and whether removing all replicas from traffic would improve the situation.

A short-lived Job may not need the same probes as an HTTP server. Choose thresholds from observed startup and recovery behavior, and test the failure case. Include the probe's own load and timeout behavior.

Test disruption and shutdown together

Replicas help only when the application can use them and they are placed across the intended failure boundaries. Required placement rules can leave pods pending when capacity is insufficient; preferred rules may allow co-location. Verify actual placement and behavior after a node or zone becomes unavailable.

A PodDisruptionBudget constrains supported voluntary evictions. It does not prevent involuntary node failure. Workload rollout controllers also have their own update settings; a PDB is not a substitute for those settings.

For three healthy replicas with minAvailable set to two, a compatible eviction can proceed while two remain healthy. If one replica is already unavailable, another eviction may block. Exercise that maintenance case before an upgrade.

Test termination with real in-flight work. The application should stop taking new work, finish or safely relinquish existing work, and exit within its supported grace period. Align load-balancer draining, worker visibility timeouts and downstream retries. A process that ignores termination can leave incomplete or duplicate actions even if Kubernetes replaces it promptly.

Verify storage recovery at the application level

A persistent volume is not automatically a backup. A snapshot may be crash-consistent rather than application-consistent. Know the driver's capabilities, storage topology, reclaim behavior and recovery dependencies.

Restore into an isolated environment and run business checks, not just a volume mount. Confirm required keys, identities, schemas and external references are available. Measure the time from recovery authorization to usable service and the amount of data lost or needing replay.

For stateful systems, test leader or quorum behavior separately from pod rescheduling. A new pod attaching storage does not establish that the application can safely accept writes.

Do not delete a volume or recovery artifact because it appears unused without checking ownership, retention and restore dependencies.

Include security evidence in the launch record

Base64 encoding is not encryption, but Kubernetes Secret storage protection depends on cluster configuration. The Secret documentation and encryption-at-rest guidance explain separate controls.

Verify authorization, storage encryption and recovery of encryption keys. An external-secret controller that copies a value into a Kubernetes Secret still leaves that value persisted through the Kubernetes API storage path. External storage alone does not establish the protection of that copy.

Check how the application consumes rotated credentials. A changed Secret does not prove a running process uses the new value. Verify workload identity, permitted network paths and the pod-security policy actually enforced on the namespace. Use the hardening checklist for targeted acceptance tests.

Treat an upgrade as a compatibility and recovery exercise

Record the supported upgrade path across control plane, nodes, controllers, CRDs, webhooks, network and storage components. Review removed APIs and workload dependencies against the version-skew policy and provider documentation.

Test the upgrade in a representative environment, including a drain blocked by a PDB and a webhook that is unavailable. Preserve a known-good application artifact, but do not confuse application rollback with control-plane downgrade.

Provider recovery options can be time-limited and version-specific. For example, review the current EKS upgrade and downgrade guidance for the selected cluster rather than assuming either that downgrade is always available or always impossible. If the supported recovery is a replacement cluster, rehearse workload and data restoration there.

Set a stop point before each irreversible step. Confirm who can authorize proceeding when a compatibility test fails.

Make the launch decision from a short evidence register

Use these exercises to close the review:

| Exercise | Acceptance evidence | | --- | --- | | Peak and rollout load | Latency or completion targets hold with expected surge and dependency behavior | | Pod and node failure | Placement, capacity and retry behavior preserve the required service | | Drain and shutdown | In-flight work is completed or recoverably handed over | | Restore | Business checks pass within the agreed recovery objectives | | Credential rotation | New credentials work and the application stops using revoked ones | | Upgrade rehearsal | Compatibility, stop conditions and supported recovery are demonstrated |

Name an owner and expiry or revisit condition for every exception. Launching with an accepted limitation can be reasonable; hiding it behind a “production-ready” label is not.

Bring the workload profile, exercise results and unresolved exceptions to Ampity's reliability review when the team needs an independent assessment. The useful deliverable is a clear launch decision and remediation order, not a universal checklist score.