Incident Response Playbook for Production Engineering Teams

A working incident response system for engineering teams, with severity rules, command roles, communication templates, recovery gates, evidence capture, post-incident...

trigger="A production event threatens customer impact, data integrity, security, regulatory obligations or a stated service objective." owner="The incident-response program owner maintains the system. During an incident, the assigned incident commander owns coordination." participants={["Incident commander", "Operations lead", "Communications lead", "Service owner", "Security or privacy lead when needed"]} prerequisites={[ "An authoritative service inventory with owners and escalation paths", "Named communication channels and a status-page process", "Access to logs, metrics, traces, deploy history and feature controls", "A tested method to page responders without depending on the affected service" ]} outputs={[ "A timestamped incident record and impact statement", "An owned mitigation and recovery plan", "Consistent internal and customer communications", "A reviewed learning record with tracked corrective actions" ]} doneWhen={[ "Customer impact has stopped and recovery evidence is stable", "Any data, security and compliance obligations have an owner", "Temporary mitigations are recorded with an expiry or follow-up", "The learning review is complete and corrective actions are scheduled" ]} timebox="Use the response steps in real time. Complete the learning review within the team’s agreed review window after recovery." />

Use this playbook as a control system

Incident response is coordinated decision-making under uncertainty. The first objective is to reduce harm. Diagnosis matters, but a team can restore service before it knows the complete root cause. The response system must keep technical work, coordination, communication and evidence capture moving without forcing one person to do all four.

This playbook covers operational and security-relevant production incidents. Adapt severity thresholds, response commitments and notification obligations to the service, customer agreements and jurisdiction. The example timings are templates, not Ampity service-level commitments.

Every incident starts with an explicit commander, impact statement and next update time. If any of those is missing, the response is still an uncoordinated investigation.

1. Prepare before the alert

An incident process cannot repair missing ownership during an outage. Establish these controls before production:

| Control | Required record | Test | |---|---|---| | Service ownership | Primary and secondary owner, escalation path, critical dependencies | Page the path during a rehearsal | | Customer impact | User journeys, critical transactions, data and contractual boundaries | Map one synthetic failure to an impact statement | | Operational access | Dashboards, logs, traces, deployment and feature-control access | A responder outside the service team can obtain the minimum view | | Communication | Internal channel, incident document, stakeholder list, status-page process | Publish a rehearsal update without touching production | | Recovery | Rollback, traffic control, failover, restore and reconciliation procedures | Exercise at least one recovery path | | Evidence | Clock source, timeline convention and durable incident record | Reconstruct a rehearsal from the record alone |

2. Declare the incident

Any responder may request an incident declaration. The incident commander confirms it and creates the record. Do not wait for a root cause. Record what is known, what remains uncertain and why coordinated response is warranted.

Declaration template

Incident ID:
Declared at (UTC):
Commander:
Service or journey affected:
Observed customer impact:
Known start time or earliest evidence:
Current severity and reason:
Data/security concern: yes / no / unknown
Current mitigation objective:
Next internal update:
Next external update, if applicable:

Severity is an impact decision

Define severity from impact, exposure and urgency. Do not use architecture symptoms such as CPU level as the severity itself.

| Level | Example impact | Response shape | Update template | |---|---|---|---| | Critical | Widespread unavailable service, active data loss, confirmed material security event or safety risk | Immediate command structure, executive/security involvement and continuous technical response | Organization-approved critical cadence | | Major | Important journey impaired for a meaningful customer segment, material latency or a credible data/security risk | Named commander and leads, prioritized mitigation and scheduled updates | Organization-approved major cadence | | Moderate | Limited degradation with a viable workaround and no evidence of material data/security impact | Service owner leads with an incident record and escalation trigger | At milestones or material change | | Minor | Small, contained issue that does not need coordinated response | Normal work management with linked evidence | Team convention |

The accountable service owner approves the final severity policy. Reassess severity whenever scope, duration, data risk or recovery confidence changes.

3. Establish command roles

Google’s incident-management guidance separates command, operations and communication so the technical lead can focus on mitigation while another person maintains coordination. Small teams may combine roles, but they must say which responsibilities are combined.

| Role | Owns | Must not become | |---|---|---| | Incident commander | Objectives, severity, role assignments, decision cadence and escalation | The only person debugging | | Operations lead | Technical investigation, mitigation plan and validation | The customer spokesperson | | Communications lead | Internal and external updates, stakeholder questions and next-update time | A speculative root-cause narrator | | Scribe | Timeline, decisions, commands, owners and evidence links | A passive transcript without decisions | | Subject specialist | A bounded investigation or change | A parallel commander |

If the commander starts hands-on debugging, explicitly transfer command first. A handover includes current impact, active objectives, role owners, risky changes in progress and the next communication deadline.

4. Stabilize before optimizing

Choose a mitigation objective that can be verified quickly. Examples include stopping a damaging write path, reducing load, isolating a failed dependency, reverting a compatible release or routing a limited workload to a healthy region.

"type": "flow", "title": "One mitigation, one observable recovery decision", "steps": [ ], "caption": "If a recovery gate fails, return to the mitigation decision with fresh evidence. Repeating the same change without reconciling its effects can increase harm." }} />

For every consequential action, record:

  • Hypothesis: what mechanism might explain the symptom?
  • Change: what exact action will be taken?
  • Owner: who is responsible for the action and validation?
  • Expected signal: what should change, where and by when?
  • Abort condition: what observation stops or reverses the action?
  • Recovery consequence: could rollback repeat side effects or corrupt state?

Rollback is not automatically safe. Schema changes, queued work, external calls and irreversible data writes may require a fix-forward or reconciliation plan even when application traffic can be returned to an older version.

| Failed mitigation or recovery gate | Operations owner response | Evidence needed before continuing | |---|---|---| | Older application cannot read the new schema | Stop rollout and use the compatibility or fix-forward plan | Read/write compatibility test against current data | | New primary has accepted writes | Fence competing writers; do not redirect to a stale former primary | Authoritative write position, reconciliation result and approved failback plan | | Requests timed out after a payment or external write | Query the authoritative system by operation identity before retrying | Confirmed effect, no effect, or an explicitly owned unknown outcome | | Failover target lacks capacity or a dependency | Contain demand or enter approved degraded mode | Target readiness, quota and user-journey checks | | Telemetry is unavailable | Use independent probes and customer evidence; state uncertainty | Enough independent evidence to justify the next action |

For a suspected security incident, involve the security lead early. Preserve relevant access logs, affected resource identities and, where feasible, volatile evidence before destructive remediation. Do not delay urgent containment when it would allow further harm. Record who captured evidence, when, where it is stored and who can access it. Keep secrets and unnecessary personal data out of the shared incident channel. NIST's incident-response guidance supports integrating response with the organization's wider risk-management process.

5. Communicate without guessing

An update should answer five questions: what users experience, when it began, what the team is doing, what users should do and when the next update will arrive. Separate confirmed impact from investigation hypotheses.

Customer update template

We are investigating [observable customer impact] affecting [scope if known].
The issue began at approximately [time, or “the start time is still being established”].
Our team is [current mitigation activity stated without speculation].
[Workaround or customer action, only when verified.]
We will provide the next update by [time], or sooner if the situation changes materially.

Internal update template

Severity / status:
Impact and scope:
Changes since the last update:
Current mitigation and owner:
Evidence supporting or contradicting the active hypothesis:
Data/security/compliance status:
Decision needed or blocker:
Next update:

Do not publish a cause until evidence supports it. Do not write “monitoring” without naming the recovery signals and observation window.

Assign privacy, legal and contractual notification assessment to the authorized owner. An engineering incident severity does not determine whether a legal notification duty exists or when its clock begins. Record the applicable obligations, decision time, approver and evidence separately from the public status update. A technical recovery announcement does not close those obligations.

6. Keep one operational timeline

Use one time zone, normally UTC, and one durable record. Capture observations and decisions, not every chat message.

| Time | Type | Record | |---|---|---| | 14:02 | Signal | Checkout success rate dropped below the service objective | | 14:05 | Decision | Declared major incident; commander and operations lead assigned | | 14:11 | Hypothesis | Failure correlates with release 2026.09.20.3; database saturation not observed | | 14:16 | Action | Traffic shifted away from the new release; owner and abort condition recorded | | 14:20 | Evidence | Synthetic checkout and customer success rate recovered | | 14:35 | Gate | Recovery window started; queue reconciliation still open |

The timeline should let a reviewer reconstruct the incident without relying on memory.

7. Prove recovery

Stopping the alert is not enough. Declare recovery only after the affected customer journey, system state and deferred work have been checked.

"The original customer-facing symptom has stopped across the affected scope", "Error, latency, saturation and dependency signals are stable for the agreed observation window", "Queued, retried or partially completed work has been reconciled", "Data integrity checks have passed or a separate remediation incident is owned", "Security, privacy and notification obligations have been assessed", "Temporary capacity, feature flags, access grants and routing changes have owners and expiry dates", "The customer and internal closing updates state what is resolved and what remains open" ]} />

If evidence is mixed, downgrade only when the remaining risk is understood and owned. Do not close the incident to improve the metric.

8. Measure the timeline consistently

Define each timestamp before calculating an average:

  • Incident start: the earliest supported time customer or system impact began.
  • Detection: the time a system or person identified the abnormal condition.
  • Acknowledgment: the time a responder accepted ownership.
  • Declaration: the time coordinated incident management began.
  • Mitigation: the time customer impact materially reduced or stopped.
  • Recovery: the time recovery gates passed.

For an incident starting at 14:00, detected at 14:04, acknowledged at 14:07, mitigated at 14:22 and recovered at 14:40:

| Measure | Calculation | Result | |---|---|---| | Time to detect | 14:04 minus 14:00 | 4 minutes | | Time to acknowledge | 14:07 minus 14:04 | 3 minutes | | Time to mitigate | 14:22 minus 14:00 | 22 minutes | | Time to recover | 14:40 minus 14:00 | 40 minutes |

Record uncertainty when the start time is estimated. Trends are useful only when teams use the same definitions and do not manipulate declaration or closure times.

9. Run the learning review

Begin with the impact and timeline. Explain contributing conditions across software, architecture, change controls, observability, documentation, staffing and decision-making. “Human error” is a starting point for investigation, not a useful cause.

Review structure

  1. Impact and duration, with the affected journeys and data boundaries.
  2. Detection and declaration, including what delayed either step.
  3. A factual timeline of decisions, actions and observed results.
  4. Contributing conditions and why existing controls did not contain them.
  5. What helped mitigation, communication and recovery.
  6. Corrective actions with owner, priority, evidence and due date.
  7. Lessons that should change another service, standard or rehearsal.

Corrective actions should change a control, not merely tell a person to be more careful. Useful actions include a new isolation boundary, a safer deployment gate, a tested recovery path, a removed single point of failure or an alert tied to a customer symptom.

10. Improve alerts without deleting coverage blindly

When an alert fires without requiring action, review it. One noisy event does not justify deleting the signal.

Use this sequence:

  1. Identify the customer or system risk the alert was meant to protect.
  2. Check whether another signal covers that risk with better precision.
  3. Review threshold, duration, grouping, dependency and maintenance behavior.
  4. Test the revised alert against historical incidents and a controlled failure.
  5. Keep a monitored fallback while confidence is built.
  6. Remove the old signal only after the owner accepts the residual risk.

11. Rehearse the process

Quarterly is not a universal cadence. Choose a cadence based on service criticality, staff turnover and rate of architectural change. Rehearse scenarios that exercise coordination and recovery, not only diagnosis.

| Rehearsal | Evidence to collect | Pass condition | Unresolved risk to record | |---|---|---|---| | Dependency outage | Dependency map, fallback signals and customer journey | Impact is contained within the approved degraded mode | Dependencies and traffic patterns not exercised | | Bad release | Release identity, traffic controls and data compatibility | Mitigation works without corrupting state | Writes or migrations that cannot be reversed | | Regional failure | Capacity, replication, routing and residency evidence | The chosen recovery objectives are demonstrated | Capacity assumptions and remaining recovery limits | | Compromised credential | Revocation, audit trail and access-path review | Access is contained and exposure assessment has an owner | Unknown access, replacement work and notification decisions |

Record the rehearsal conditions. A passing exercise does not prove recovery under a larger outage or an untested dependency failure.

12. Preserve security and forensic evidence when required

An availability incident can also involve unauthorized access, data exposure or deliberate activity. The incident commander should know when to involve the security owner and when ordinary recovery actions could destroy evidence. Restarting hosts, rotating logs or deleting suspicious resources may improve availability while weakening the investigation.

Use the organization’s security incident procedure for evidence preservation, access containment, notification decisions and legal obligations. Restrict sensitive incident artifacts to the people who need them. A broad incident channel is not an appropriate location for credentials, personal data or unrestricted forensic exports.

Separate facts from access-controlled investigative hypotheses in customer communication. Do not delay a necessary availability update until every security question is resolved, but do not declare that no exposure occurred without evidence and authorized review.

13. Handover without losing command state

Long incidents need deliberate handover. The outgoing commander records current impact, confirmed facts, active hypotheses, actions in progress, stop conditions, unresolved approvals and the next communication deadline. The incoming commander acknowledges the transfer and names any gap before assuming authority.

Keep one operational timeline across shifts. Parallel private notes create conflicting histories. Restrict sensitive details through linked controlled records while preserving the shared decision sequence.

After recovery, hand pending reconciliation, customer follow-up and corrective actions to named owners. Closing the incident channel does not close uncertain transactions, delayed jobs or temporary access. Track those obligations until their acceptance evidence exists.

14. Test the communication dependency

Exercise the status page, internal coordination channel and customer-update path during a simulated primary identity or messaging outage. Confirm who can publish, how approvals work and where the current facts are sourced.

Prepare a fallback that does not require the failing service. Keep templates factual and adaptable. A template should accelerate structure, not pre-fill an unverified cause, resolution time or claim that data was unaffected.

Review whether support, sales and leadership know where to find the authoritative update. Parallel unofficial updates can create conflicting commitments and distract responders from recovery.

Retain the communication exercise result with the service readiness evidence.

Repeat it after ownership or communication-platform changes.

15. Operational readiness checklist

Ownership and access

"Every production service has a primary and secondary owner", "Escalation paths work outside normal business hours where required", "Responders can access the minimum telemetry and change controls", "Emergency access is time-bound, recorded and reviewed" ]} />

Command and communication

"Severity criteria are approved for the operating context", "Role cards and handover requirements are documented", "Internal and customer update templates have owners", "The status-page path can operate during a primary-service outage" ]} />

Recovery and learning

"At least one rollback, failover or restore path is exercised", "Recovery includes data and deferred-work reconciliation", "Learning reviews avoid blame and assign evidence-based actions", "Overdue corrective actions are visible to service leadership" ]} />

Limitations and approval boundaries

This is an operating template, not a substitute for a service-specific recovery procedure, forensic investigation or legal advice. Validate commands, access rights, backup restoration and failover behavior in your environment. The service owner approves recovery objectives and residual risk; security and legal owners approve decisions within their remit. Neither a completed checklist nor a low average recovery time certifies that the service is secure.

Keep review actions open until their acceptance evidence exists. A scheduled ticket is evidence of ownership, not evidence that the control works.

What to do this week

Pick one customer-critical service. Name the commander, operations lead and communications lead for a rehearsal. Run a thirty-minute scenario, require a declaration record and customer update, then prove one recovery gate. The gaps found in that exercise are the first incident-response backlog.

Primary references