Incident Response Playbook for Production Engineering Teams

A working incident response system for engineering teams, with severity rules, command roles, communication templates, recovery gates, evidence capture, post-incident...

Use this playbook as a control system

Incident response is coordinated decision-making under uncertainty. The first objective is to reduce harm. Diagnosis matters, but a team can restore service before it knows the complete root cause. The response system must keep technical work, coordination, communication and evidence capture moving without forcing one person to do all four.

This playbook covers operational and security-relevant production incidents. Adapt severity thresholds, response commitments and notification obligations to the service, customer agreements and jurisdiction. The example timings are templates, not Ampity service-level commitments.

Every incident starts with an explicit commander, impact statement and next update time. If any of those is missing, the response is still an uncoordinated investigation.

1. Prepare before the alert

An incident process cannot repair missing ownership during an outage. Establish these controls before production:

2. Declare the incident

Any responder may request an incident declaration. The incident commander confirms it and creates the record. Do not wait for a root cause. Record what is known, what remains uncertain and why coordinated response is warranted.

Declaration template

Incident ID:
Declared at (UTC):
Commander:
Service or journey affected:
Observed customer impact:
Known start time or earliest evidence:
Current severity and reason:
Data/security concern: yes / no / unknown
Current mitigation objective:
Next internal update:
Next external update, if applicable:

Severity is an impact decision

Define severity from impact, exposure and urgency. Do not use architecture symptoms such as CPU level as the severity itself.

The accountable service owner approves the final severity policy. Reassess severity whenever scope, duration, data risk or recovery confidence changes.

3. Establish command roles

Google’s incident-management guidance separates command, operations and communication so the technical lead can focus on mitigation while another person maintains coordination. Small teams may combine roles, but they must say which responsibilities are combined.

If the commander starts hands-on debugging, explicitly transfer command first. A handover includes current impact, active objectives, role owners, risky changes in progress and the next communication deadline.

4. Stabilize before optimizing

Choose a mitigation objective that can be verified quickly. Examples include stopping a damaging write path, reducing load, isolating a failed dependency, reverting a compatible release or routing a limited workload to a healthy region.

For every consequential action, record:

• Hypothesis: what mechanism might explain the symptom?
• Change: what exact action will be taken?
• Owner: who is responsible for the action and validation?
• Expected signal: what should change, where and by when?
• Abort condition: what observation stops or reverses the action?
• Recovery consequence: could rollback repeat side effects or corrupt state?

Rollback is not automatically safe. Schema changes, queued work, external calls and irreversible data writes may require a fix-forward or reconciliation plan even when application traffic can be returned to an older version.

For a suspected security incident, involve the security lead early. Preserve relevant access logs, affected resource identities and, where feasible, volatile evidence before destructive remediation. Do not delay urgent containment when it would allow further harm. Record who captured evidence, when, where it is stored and who can access it. Keep secrets and unnecessary personal data out of the shared incident channel. NIST's incident-response guidance supports integrating response with the organization's wider risk-management process.

5. Communicate without guessing

An update should answer five questions: what users experience, when it began, what the team is doing, what users should do and when the next update will arrive. Separate confirmed impact from investigation hypotheses.

Customer update template

We are investigating [observable customer impact] affecting [scope if known].
The issue began at approximately [time, or “the start time is still being established”].
Our team is [current mitigation activity stated without speculation].
[Workaround or customer action, only when verified.]
We will provide the next update by [time], or sooner if the situation changes materially.

Internal update template

Severity / status:
Impact and scope:
Changes since the last update:
Current mitigation and owner:
Evidence supporting or contradicting the active hypothesis:
Data/security/compliance status:
Decision needed or blocker:
Next update:

Do not publish a cause until evidence supports it. Do not write “monitoring” without naming the recovery signals and observation window.

Assign privacy, legal and contractual notification assessment to the authorized owner. An engineering incident severity does not determine whether a legal notification duty exists or when its clock begins. Record the applicable obligations, decision time, approver and evidence separately from the public status update. A technical recovery announcement does not close those obligations.

6. Keep one operational timeline

Use one time zone, normally UTC, and one durable record. Capture observations and decisions, not every chat message.

The timeline should let a reviewer reconstruct the incident without relying on memory.

7. Prove recovery

Stopping the alert is not enough. Declare recovery only after the affected customer journey, system state and deferred work have been checked.

• The original customer-facing symptom has stopped across the affected scope
• Error, latency, saturation and dependency signals are stable for the agreed observation window
• Queued, retried or partially completed work has been reconciled
• Data integrity checks have passed or a separate remediation incident is owned
• Security, privacy and notification obligations have been assessed
• Temporary capacity, feature flags, access grants and routing changes have owners and expiry dates
• The customer and internal closing updates state what is resolved and what remains open

If evidence is mixed, downgrade only when the remaining risk is understood and owned. Do not close the incident to improve the metric.

8. Measure the timeline consistently

Define each timestamp before calculating an average:

• Incident start: the earliest supported time customer or system impact began.
• Detection: the time a system or person identified the abnormal condition.
• Acknowledgment: the time a responder accepted ownership.
• Declaration: the time coordinated incident management began.
• Mitigation: the time customer impact materially reduced or stopped.
• Recovery: the time recovery gates passed.

For an incident starting at 14:00, detected at 14:04, acknowledged at 14:07, mitigated at 14:22 and recovered at 14:40:

Record uncertainty when the start time is estimated. Trends are useful only when teams use the same definitions and do not manipulate declaration or closure times.

9. Run the learning review

Begin with the impact and timeline. Explain contributing conditions across software, architecture, change controls, observability, documentation, staffing and decision-making. “Human error” is a starting point for investigation, not a useful cause.

Review structure

1. Impact and duration, with the affected journeys and data boundaries.
2. Detection and declaration, including what delayed either step.
3. A factual timeline of decisions, actions and observed results.
4. Contributing conditions and why existing controls did not contain them.
5. What helped mitigation, communication and recovery.
6. Corrective actions with owner, priority, evidence and due date.
7. Lessons that should change another service, standard or rehearsal.

Corrective actions should change a control, not merely tell a person to be more careful. Useful actions include a new isolation boundary, a safer deployment gate, a tested recovery path, a removed single point of failure or an alert tied to a customer symptom.

10. Improve alerts without deleting coverage blindly

When an alert fires without requiring action, review it. One noisy event does not justify deleting the signal.

Use this sequence:

1. Identify the customer or system risk the alert was meant to protect.
2. Check whether another signal covers that risk with better precision.
3. Review threshold, duration, grouping, dependency and maintenance behavior.
4. Test the revised alert against historical incidents and a controlled failure.
5. Keep a monitored fallback while confidence is built.
6. Remove the old signal only after the owner accepts the residual risk.

11. Rehearse the process

Quarterly is not a universal cadence. Choose a cadence based on service criticality, staff turnover and rate of architectural change. Rehearse scenarios that exercise coordination and recovery, not only diagnosis.

Record the rehearsal conditions. A passing exercise does not prove recovery under a larger outage or an untested dependency failure.

12. Preserve security and forensic evidence when required

An availability incident can also involve unauthorized access, data exposure or deliberate activity. The incident commander should know when to involve the security owner and when ordinary recovery actions could destroy evidence. Restarting hosts, rotating logs or deleting suspicious resources may improve availability while weakening the investigation.

Use the organization’s security incident procedure for evidence preservation, access containment, notification decisions and legal obligations. Restrict sensitive incident artifacts to the people who need them. A broad incident channel is not an appropriate location for credentials, personal data or unrestricted forensic exports.

Separate facts from access-controlled investigative hypotheses in customer communication. Do not delay a necessary availability update until every security question is resolved, but do not declare that no exposure occurred without evidence and authorized review.

13. Handover without losing command state

Long incidents need deliberate handover. The outgoing commander records current impact, confirmed facts, active hypotheses, actions in progress, stop conditions, unresolved approvals and the next communication deadline. The incoming commander acknowledges the transfer and names any gap before assuming authority.

Keep one operational timeline across shifts. Parallel private notes create conflicting histories. Restrict sensitive details through linked controlled records while preserving the shared decision sequence.

After recovery, hand pending reconciliation, customer follow-up and corrective actions to named owners. Closing the incident channel does not close uncertain transactions, delayed jobs or temporary access. Track those obligations until their acceptance evidence exists.

14. Test the communication dependency

Exercise the status page, internal coordination channel and customer-update path during a simulated primary identity or messaging outage. Confirm who can publish, how approvals work and where the current facts are sourced.

Prepare a fallback that does not require the failing service. Keep templates factual and adaptable. A template should accelerate structure, not pre-fill an unverified cause, resolution time or claim that data was unaffected.

Review whether support, sales and leadership know where to find the authoritative update. Parallel unofficial updates can create conflicting commitments and distract responders from recovery.

Retain the communication exercise result with the service readiness evidence.

Repeat it after ownership or communication-platform changes.

15. Operational readiness checklist

Ownership and access

• Every production service has a primary and secondary owner
• Escalation paths work outside normal business hours where required
• Responders can access the minimum telemetry and change controls
• Emergency access is time-bound, recorded and reviewed

Command and communication

• Severity criteria are approved for the operating context
• Role cards and handover requirements are documented
• Internal and customer update templates have owners
• The status-page path can operate during a primary-service outage

Recovery and learning

• At least one rollback, failover or restore path is exercised
• Recovery includes data and deferred-work reconciliation
• Learning reviews avoid blame and assign evidence-based actions
• Overdue corrective actions are visible to service leadership

Limitations and approval boundaries

This is an operating template, not a substitute for a service-specific recovery procedure, forensic investigation or legal advice. Validate commands, access rights, backup restoration and failover behavior in your environment. The service owner approves recovery objectives and residual risk; security and legal owners approve decisions within their remit. Neither a completed checklist nor a low average recovery time certifies that the service is secure.

Keep review actions open until their acceptance evidence exists. A scheduled ticket is evidence of ownership, not evidence that the control works.

What to do this week

Pick one customer-critical service. Name the commander, operations lead and communications lead for a rehearsal. Run a thirty-minute scenario, require a declaration record and customer update, then prove one recovery gate. The gaps found in that exercise are the first incident-response backlog.

Primary references

• NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management, published April 2025.
• Google SRE: Incident Management Guide.
• Google SRE: Managing Incidents.
• Google SRE Workbook: Postmortem Culture.