Kubernetes Production Operations: The Change and Recovery Evidence Pack

Establish who operates a Kubernetes service, what must survive disruption, and how upgrades and recovery are tested. Includes a version matrix, restore contract,...

Decision brief

A healthy cluster is not evidence that an application can survive a failed change or recover its data. Production operations should be accepted against a service-specific evidence pack: ownership, supported component versions, measured failure behavior, independently accessible recovery assets, and a rehearsed return to useful service.

This whitepaper is for platform engineers, SREs, and application owners preparing a Kubernetes workload for ongoing operation or a material upgrade. Its thesis is that change and recovery are coupled responsibilities, but they are not the same operation. Reverting a manifest can change desired configuration. It does not reverse a database migration or recover missing records.

The scope is an operational acceptance method, not a comprehensive hardening standard or a managed-service promise. The scenarios and templates are illustrative. There are no customer audit, availability, cost, or deployment-frequency results. This paper remains reviewed, non-indexable, and factually unapproved pending named technical review.

1. Define the service, not an enterprise benchmark

Start with one user journey and the dependencies required to complete it. For an illustrative processing API, “accept a request” and “finish the requested processing” are different outcomes. Define which one the availability indicator measures, how delayed work is reported, and what data can be lost.

Agree recovery time and recovery point objectives with the service owner. Record the event that starts each clock and the evidence that stops it. A running pod, a successful deployment, and a consistent restored business record are three different observations.

Do not import universal replica counts, uptime percentages, recovery tiers, node sizes, or upgrade cadences from a generic guide. Select tolerances from workload behavior, dependency constraints, support coverage, and the cost of failure. An untested objective remains a target, not a demonstrated capability.

2. Create a responsibility and dependency ledger

Managed Kubernetes changes who operates parts of the control plane. It does not remove application, data, identity, network, and integration responsibilities. Document the actual provider service boundary and escalation process for the selected offering.

| Boundary | Accountable role to assign | Evidence required before acceptance | | --- | --- | --- | | Provider control plane | Internal cloud owner and provider support contact | Service scope, access route, version lifecycle, recovery process | | Nodes and add-ons | Platform owner | Image lifecycle, runtime, CNI/CSI compatibility, replacement rehearsal | | Application behavior | Service owner | Probes, shutdown, retry, overload, and compatibility tests | | Persistent data | Data owner | Consistent backup, restore, retention, and reconciliation evidence | | Identity and secrets | Security/identity owner | Least privilege, key recovery, revocation, emergency access | | Incident coordination | Named incident lead | Paging route, authority to stop changes, escalation and handoff |

One person may hold several roles in a small team. What matters is acknowledged responsibility and realistic coverage. A document cannot create an on-call service or authorize a third party to change production.

3. Inventory the supported operating envelope

Capture the control-plane version, node OS/image, container runtime, kubelet, network and storage plugins, DNS, ingress/gateway controller, admission components, autoscaler, and backup tooling. Include configuration flags and important custom resources, not just product names.

For each component, record its installed version, intended target, support end date, source of lifecycle information, compatibility requirements, and rollback or replacement method. Date the check. A maintained project can still have an unsupported installed version.

Use the upstream version skew policy together with provider and add-on requirements. Skew allowances differ by component; they are not permission to skip an arbitrary upgrade path. Validate removed APIs and conversion webhooks using the actual manifests and live resources.

Avoid an evergreen OS shortlist. A node image must be supported for the selected Kubernetes version and provider, with an explicit patch and replacement owner. Test image changes against storage mounts, networking, startup, architecture-specific dependencies, and security agents before general rollout.

4. Correct the ingress lifecycle assumption

The Kubernetes project's November 2025 announcement scheduled community ingress-nginx maintenance to end in March 2026, with no subsequent releases or security fixes. It should not be presented as a maintained default for a new production design. This refers to that community project, not every product using NGINX. See the official retirement notice.

Inventory the exact controller and image before drawing a conclusion from its name. A migration candidate may use Gateway API or another supported controller, but API support alone does not establish equivalent behavior.

Build a route-parity packet covering TLS and certificate renewal, authentication integration, rewrites, headers, timeouts, body limits, long-lived connections, and rate controls. Test in a separate environment and then with an approved traffic boundary. Preserve a recovery route only if it remains safe and supported; do not treat indefinite return to an unmaintained component as the long-term plan.

5. Separate desired configuration, access, and data recovery

A Git repository can preserve declared configuration. It is not inherently a backup of application data, encryption keys, external secrets, runtime-created resources, registry artifacts, or the control plane's complete state.

For self-managed clusters, etcd recovery has its own snapshot, key, membership, and version requirements. Follow Kubernetes etcd operations guidance and rehearse the selected procedure. An etcd snapshot does not contain the bytes in application persistent volumes.

For managed services, identify what recovery the provider exposes and what it does not. Do not assume direct etcd access or that a provider's internal backup is an application restore interface available to your operators.

6. Write a restore contract for each stateful dependency

A snapshot's existence is not proof of application consistency. Determine whether the database requires quiescing, transaction-log replay, coordinated snapshots, or an application-aware backup procedure. Include externally managed databases and object stores in the dependency map.

Record:

  • Authoritative dataset and owner, including external dependencies.
  • Backup method, recoverable timestamp, retention, and corruption detection.
  • Credentials and decryption keys available independently of the source cluster.
  • Restore destination, isolation controls, and application-version compatibility.
  • Reconciliation queries or business invariants that establish usable state.
  • Accepted data loss, traffic authority, and a safe return or roll-forward plan.

Run a restore into an isolated destination. Verify that it cannot send real notifications, capture payments, or consume production work accidentally. A rehearsal should include locating assets and obtaining approved access, not start after an expert has manually prepared everything.

Report the measured timeline and its exclusions. A successful small-dataset test does not establish recovery time for the production dataset. Investigate missing records or mismatched totals before declaring the recovery complete.

7. Distinguish voluntary and involuntary disruption

A PodDisruptionBudget constrains supported voluntary eviction operations. It does not prevent node failure, zone loss, or forced termination of interruptible capacity. Kubernetes documents that distinction in its disruption guidance.

Replica count alone is insufficient. Placement, spare capacity, storage accessibility, dependency quotas, and application coordination determine what can continue. A workload may have several replicas but share one failure domain or one non-redundant database.

Test node drain separately from node loss. During an approved drain rehearsal, inspect eviction decisions, pending pods, shutdown behavior, and traffic impact. During a controlled failure exercise, measure detection and rescheduling without assuming that an eviction budget can delay the failure.

If a drain is blocked, investigate unavailable replicas and policy intent. Deleting the disruption budget is not a routine workaround. A change owner should choose among restoring health, adding capacity, rescheduling, or accepting a documented interruption.

8. Connect probes and shutdown to application semantics

Readiness determines whether a pod is ready for service traffic. Liveness can trigger restarts, and a startup probe can protect slow initialization from premature checks. Their configured behavior needs to match the application, as explained in the probe documentation.

Do not make liveness depend blindly on a shared downstream service. A dependency failure could then restart every replica and destroy useful capacity. Conversely, a readiness response that always succeeds can route requests to a pod that cannot perform its promised work.

Test warmup, dependency unavailability, connection draining, and in-flight work at termination. Document what happens when the grace period expires. A retry needs bounded backoff and an idempotency strategy where work has side effects; Kubernetes rescheduling does not supply that application contract.

9. Prove policy enforcement at the actual boundary

Start with the data flows that must be allowed and denied. Kubernetes NetworkPolicy requires a supporting network implementation, and standard NetworkPolicy does not define a universal logging-only mode. Observation or audit modes, when available, belong to a specific implementation and version. See the NetworkPolicy documentation.

Test ingress and egress independently with permitted and forbidden identities, namespaces, destinations, and ports. Include DNS, identity endpoints, backup destinations, and observability paths. A default-deny rollout can disrupt recovery if those dependencies are omitted.

Use Pod Security Standards as one workload-control baseline, with versioned enforcement and reviewed exceptions. They do not establish complete tenant isolation or audit compliance. Restrict human and workload permissions separately; avoid making every platform engineer a permanent cluster administrator.

Keep enforcement evidence and exception expiry with the workload. Record who can alter labels, policies, admission configuration, and service accounts, because a control that the wrong principal can bypass is not the intended boundary.

10. Measure CPU, memory, and failure capacity separately

Requests inform scheduling; limits constrain runtime resource use with resource-specific behavior. CPU throttling and memory exhaustion are not interchangeable failure modes. Use the official resource management guidance for the selected runtime and version.

Profile representative peaks, startup, compaction, batch activity, and dependency slowdown. A single usage percentile can hide a memory spike that terminates a process. Include system and DaemonSet overhead, placement constraints, storage limits, and the capacity needed when an agreed failure domain is unavailable.

A proposed request reduction must pass performance and disruption tests before its financial impact is assessed. Lower requests may leave the same nodes running because of minimum counts, fragmentation, topology constraints, or autoscaler policy. Existing commitments may also prevent an immediate invoice reduction.

Use a capacity ledger: observed demand, configured requests and limits, available allocatable resources, failure reserve, proposed change, test result, node consolidation actually observed, and bill impact. Do not label theoretical capacity savings as realized cost savings.

11. Test autoscaling as a delayed control loop

For the selected autoscaler, record the signal, sampling delay, decision behavior, capacity provisioning time, and readiness delay. Decide what the service does before new capacity becomes usable: queue, reject, degrade, or consume reserved headroom.

Exercise missing metrics, a saturated downstream dependency, an exhausted infrastructure quota, and unavailable instance types. More replicas can amplify a constrained database rather than improve service. Bound maximum expansion against downstream and financial limits.

Interruptible nodes need workload-specific recovery behavior and an assessed capacity fallback. Notice periods and replacement availability should not be treated as guarantees. Include repeated interruption and simultaneous demand in the rehearsal, rather than extrapolating from one successful eviction.

12. Use a provider-specific upgrade and recovery plan

Separate application rollback, node replacement, add-on recovery, and control-plane recovery. They have different compatibility constraints and authority. A Git revert may restore old configuration while the new application has already changed data.

Before upgrading, complete a version matrix with these columns: component, current version, target version, support source checked on date, allowed intermediate states, test evidence, change owner, and supported recovery method. Capture removed APIs and custom-resource conversion behavior before a control-plane transition.

Do not claim that every managed service permits or forbids a downgrade. The current Amazon EKS update documentation, for example, describes a time-limited previous-version rollback path. Eligibility, timing, component compatibility, and provider procedures must be checked for the actual cluster at change approval.

Where in-place recovery is unavailable or insufficient, evaluate a replacement cluster and workload/data migration. That option still requires capacity, identity, networking, compatible data, and a tested traffic transition. It is not an instant fallback.

13. Rehearse the failure packet before change approval

| Exercise | Observation required | Stop or investigate when | | --- | --- | --- | | Dependency outage | Bounded retries and a defined degraded response | Retry amplification or silent data loss | | Node drain | Eviction decisions and replacement readiness | Capacity or disruption policy blocks safe progress | | Node or zone loss | Workload and data behavior without graceful eviction | Surviving capacity cannot meet the agreed service target | | Backup restore | Business reconciliation and measured timeline | Keys, artifacts, records, or external dependencies are missing | | Component upgrade | Supported mixed versions and user-journey checks | Compatibility or recovery assumptions are unproven | | Policy change | Required flows work and forbidden flows fail | A required recovery path is lost or isolation is bypassed |

Run disruptive tests only in an authorized scope with an abort authority and recovery assets ready. Start in a representative test environment; production exercises require a separate risk decision. “Chaos testing” is not permission to inject failure into customer workloads.

Keep the results as an evidence packet rather than a green checklist. For every failed test, record the affected service, root cause if known, corrective action, owner, and whether the unresolved risk prevents the planned change.

14. Hand over an operable service

The acceptance pack should contain the dependency/ownership ledger, current version matrix, alert routes, policy evidence, capacity ledger, backup/restore contract, and last rehearsed change plan. Include the date and scope of each test, not just links to dashboards.

Distinguish symptoms from pages. A restart or warning event may deserve investigation without waking an operator; a violated user-journey objective may require immediate action. Define signal loss and notification-path failure so a silent monitoring system cannot look healthy.

The receiving team should demonstrate that it can find the evidence, access the required systems, and execute a bounded recovery exercise. Record unresolved exceptions and the authority that accepted them. A handoff meeting is not evidence of ongoing support capacity.

A reliability review can identify and prioritize these gaps. Bring the dependency ledger, version matrix, recent incidents, and available restore evidence. A useful review output is a remediation backlog with risk, owner, validation method, and unresolved assumptions for each item. Cloud platform engineering can address repeated provisioning or operating constraints. Any implementation or ongoing operations scope needs a separately agreed boundary, access model, and owner.

15. Make operational and security authority explicit

Kubernetes concentrates powerful change mechanisms behind APIs, controllers and credentials. Production readiness therefore depends on who can change desired state, who can change the controls governing that state, and how emergency authority is bounded. A namespace role, cloud role, Git permission and delivery-system permission can each provide a different path to production. Review them as one authority graph.

Begin with human access. Map the routine roles required by application operators, platform operators, security responders and auditors. Separate read, deploy, policy, secret and cluster-administration authority. Require stronger, time-bounded approval for emergency access, and test the complete path before an incident. A break-glass account whose credential cannot be recovered, whose audit trail is inaccessible, or whose role is too broad is not an adequate control.

Then map workload identity. Each controller, operator and application service account should have a stated purpose and resource boundary. Test whether a compromised pod can create workloads, read unrelated secrets, alter admission policy, impersonate another identity, or reach cloud APIs beyond its need. Namespace separation does not by itself establish a tenant boundary. The selected network, runtime, node, identity and admission controls need a combined threat assessment.

Protect the systems that apply desired state. A pull request approval is ineffective if a user can bypass it through direct cluster or cloud access. Conversely, removing all direct access can make recovery impossible when the delivery system is unavailable. Define an emergency change route with identity verification, limited scope, retained commands or API records, peer review where feasible, and mandatory reconciliation back into the declared configuration.

Secrets require lifecycle evidence. Record their source, delivery mechanism, authorized readers, rotation owner, revocation behavior and recovery dependency. Avoid copying secret values into manifests, operation logs, diagnostic bundles or backup reports. Test rotation while workloads are active, including how long old credentials remain accepted and how a failed rollout returns to a valid state.

Audit evidence must remain available when the cluster is degraded. Export the necessary control-plane, identity, admission and cloud activity records to a protected destination with an explicit retention policy. Verify that timestamps, actor identity, request targets and outcomes can be correlated. Logging every object body may expose sensitive data, so configure scope and access deliberately.

Security changes use the same staged operating model as application changes. A new policy can block DNS, image retrieval, telemetry or backup traffic. Start with a representative environment, prove intended allows and denies, deploy to a bounded cohort, and retain an authorized recovery path that does not silently remove the control from every workload.

16. Rehearse a failed upgrade from alert to useful service

A useful rehearsal follows the operator's actual decision sequence. Consider a node-image upgrade in which replacement nodes join, but a storage or networking incompatibility leaves part of the service unavailable. The exercise begins with the alert and user symptom, not with a prepared diagnosis.

The incident lead first stops further expansion and identifies the affected cohort. The platform operator checks node, add-on, scheduling and network evidence. The application owner verifies the user journey and distinguishes rejected, delayed and accepted work. The data owner determines whether any writes are missing, duplicated or awaiting reconciliation. The security owner confirms that emergency access and any temporary route preserve required controls.

The team then chooses among restoring compatible nodes, moving workloads to an unaffected pool, replacing the cluster, or applying a forward fix. That choice depends on provider support, version compatibility, storage attachment, available capacity and data state. “Roll back” is not a sufficient instruction. Record the exact object or system being restored and the point beyond which reversal is unsafe.

During the exercise, retain these timestamps:

  • first user impact and first reliable signal;
  • change stop and cohort containment;
  • diagnosis sufficient to select a recovery path;
  • capacity available for the selected path;
  • application readiness and dependency recovery;
  • business-data reconciliation complete; and
  • service owner acceptance of useful service.

These timestamps answer different questions. Pods becoming Ready can precede backlog recovery or business reconciliation. The declared recovery time should end at the accepted business condition, with exclusions documented. Do not remove difficult intervals from the measurement because they belong to another team.

Test the communications path too. The incident lead needs an owner directory, current provider escalation route, change authority and a method for communicating uncertainty. Support should know what can be said when a request was accepted but its final effect is unknown. Avoid reporting recovery while users continue to receive stale or inconsistent results.

After recovery, compare the system with the pre-change state. Confirm node and add-on versions, admission behavior, network policy, identity bindings, storage attachments, autoscaling, alert routes and backup schedules. Remove temporary access and capacity with named owners. Reconcile direct emergency changes into version-controlled desired state. Retain the incident evidence with the next upgrade decision.

The rehearsal is successful when it exposes a correctable gap and the team closes or explicitly accepts it. A perfectly scripted exercise in which every dependency behaves as expected provides limited evidence. Introduce one controlled ambiguity, such as an incomplete provider response or delayed storage attachment, and observe whether authority and evidence remain clear.

17. Reference flow for change, containment and recovery

18. Operational acceptance checklist and next action

"The service boundary, user journey, dependencies, recovery objectives and data-loss tolerance are approved by accountable owners.", "The installed control plane, node, runtime, network, storage, ingress, admission, autoscaling and backup versions have current support evidence.", "Human, workload, delivery-system and emergency authority paths are mapped and tested as one access graph.", "Desired configuration, application data, artifacts, keys and recovery identity can be recovered independently of the failed cluster.", "Probes, shutdown, retries and overload behavior match application semantics and preserve external-effect safety.", "Voluntary disruption, node loss, dependency failure, policy change and restore exercises have dated evidence and named owners.", "The upgrade plan defines compatible intermediate states, bounded cohorts, abort conditions and the point where reversal becomes unsafe.", "Audit, security and observability evidence remain accessible during control-plane or delivery-system failure.", "A receiving operator can diagnose, contain and recover a representative failure without undocumented specialist knowledge.", "Service acceptance follows business reconciliation, removal of temporary controls and explicit residual-risk approval." ]} />

The practical next step is to choose one material upcoming change and build the evidence pack around it. Run the restore and failure exercises before the change window, not during it. If the required owners, recovery assets or compatible path are missing, postpone the change or narrow its scope. Production readiness is the demonstrated ability to make and recover from a change within the agreed service boundary.

19. Preserve a reusable change decision record

The decision record links the approved change to the evidence available at that time. Capture the service and cluster identifiers, accountable owners, current and target component versions, intended outcome, affected workload class, data implications, provider constraints, known unsupported states, rollout cohort, stop conditions and recovery options. Include links to the exact manifests, artifacts and test results rather than a mutable dashboard home page.

Record assumptions in a form that can expire. A storage driver may be compatible only with a particular node image. A control-plane recovery option may exist only inside a provider-defined time window. Capacity may be available during the rehearsal but not during a seasonal peak. Give each material assumption an evidence source, check date and owner who must revalidate it at approval.

Separate authorization from execution. The approver accepts the stated risk and scope. The change owner coordinates the rollout. The platform operator performs infrastructure actions. The service and data owners decide whether user behavior and business records are acceptable. One person may fill multiple roles, but the record should show which decision they made and when.

During the change, append observations rather than rewriting the original plan. Note cohort start and end, actual versions, unexpected warnings, pauses, emergency changes and deviations from the approved sequence. This preserves the difference between what was expected and what occurred. If a stop condition is waived, record the new authority and reasoning before continuing whenever the situation permits.

At closure, state whether the intended outcome was achieved, whether any service or data objective was violated, what temporary capacity or access remains, which follow-up work is required and when the next lifecycle review occurs. Link the final reconciliation evidence and receiving-owner acceptance. A successful technical apply with unresolved business exceptions remains an open change.

Use the record in the next upgrade. Compare estimates with actual timings, support effort, failure behavior and recovery limitations. Update the standard packet only where the evidence justifies the change. Do not turn one smooth upgrade into a universal assumption, and do not hide a difficult upgrade by replacing the historical record with a cleaner template.

Retain the record according to the service's audit and incident-learning policy. Restrict sensitive infrastructure details, identity data and secrets while keeping enough provenance for an authorized reviewer to reconstruct the decision. Evidence that nobody can retrieve during an incident or later review does not support the operating claim.

Limitations and approval evidence

This paper does not certify a cluster, establish a regulatory control set, promise availability, or supply a universal production configuration. It deliberately excludes unsupported anonymous migration results and finance-sector audit claims.

Before publication approval, obtain a named Kubernetes/SRE reviewer, verify the dated component references, and review the restore, network, and upgrade contracts for technical accuracy. Before operational acceptance, attach workload-specific tests, provider/version eligibility, approved disruption scope, measured recovery evidence, and a receiving owner.

Primary references

Source check: 20 September 2026. This is a documentation check, not named human technical approval. Recheck provider and component eligibility when approving a change.