Incident Management: A Safe Response Playbook

Define incident authority, choose safe mitigations, communicate uncertainty, and verify recovery with a worked timeline and reusable response checklist.

A useful incident playbook helps a responder decide what to do while information is incomplete. It identifies who can act, what must be protected, which actions are reversible and how to tell whether customers have recovered.

The objective is not simply to close an alert. Restoring new requests, correcting affected records and preserving evidence may be separate workstreams. A service can look healthy again while customers still have missing or duplicate business actions.

This guide gives a response structure and an illustrative exercise. Adapt it to the service's risks, staffing and recovery requirements. It is not a promise that any organization can provide a particular response time without the people and authority to support it.

Decide what deserves an incident

Use customer impact and risk to determine severity, not the loudest dashboard. A broad outage, a correctness failure affecting a small number of financial records, and suspected unauthorized access may require different specialist responses.

Write severity criteria in terms responders can observe:

| Condition | Response to define locally | | --- | --- | | Critical user journey unavailable | Who leads, who joins, and how customers receive updates | | Data loss or incorrect business action | Who can stop writes, preserve records and authorize repair | | Security concern | How security joins and evidence is protected | | Degraded service with a workaround | How impact is tracked and when escalation is required | | Internal warning without impact | Whether investigation can remain a ticket rather than a page |

An incident may be declared before its cause or full scope is known. Allow responders to revise severity as evidence changes. Do not penalize an early declaration that helped organize a credible risk.

Define the acknowledgement and escalation path from the actual coverage model. A round-the-clock service commitment requires a staffed and sustainable arrangement, not just a paging tool.

Page for action, not every deviation

A page should identify an actionable condition, the affected service, a runbook and an accountable responder. If nobody can do anything useful before the next working period, a ticket may be more appropriate.

The Google SRE Workbook's SLO alerting chapter discusses alerting against service objectives and error-budget consumption. Use the method with your own objectives, traffic and risk tolerance. Its examples are not universal thresholds for every application.

Low traffic, delayed batch processing and correctness failures need additional care. A percentage-based availability signal can be quiet when a scheduled export never ran. Monitor completion and business invariants where they matter.

Review duplicate pages, false positives and missed incidents with the on-call team. Track interruptions and recovery time for people as well as systems. If the same unresolved condition pages every shift, fix the alert or underlying problem rather than treating the volume as normal.

Establish authority early

The Google SRE incident-response guidance emphasizes organized command, operational work and communication. Use named responsibilities even if a small incident has one person wearing several roles.

| Role | Responsibility | | --- | --- | | Incident commander | Maintains priorities, delegates work, resolves competing actions and approves the response direction | | Operations lead | Coordinates diagnosis and changes, with clear change owners | | Communications lead | Publishes verified impact, uncertainty and the next update time | | Scribe | Records timestamps, evidence, hypotheses, actions and results |

The commander does not need to be the deepest technical expert. They need enough distance to notice that several teams are making conflicting changes or that nobody is checking customer impact.

Use one incident record and a known communication channel. Record the current commander and make handovers explicit: the incoming person confirms ownership, current impact, active changes and unresolved risks. Preserve an out-of-band contact path for failure of the normal collaboration system.

Use a safe-mitigation decision gate

“Rollback first” is unsafe when the previous binary cannot read the current data or when the bad release already caused external effects. A rollback can restore code without undoing a payment, message or schema migration.

Before any material action, identify its target, expected effect, owner, evidence to preserve and stop condition. Emergency authority should be pre-agreed so the check is quick enough to use.

| Candidate action | Gate before execution | | --- | --- | | Roll back code | Prior version is compatible with current schema, data and dependencies; artifact and procedure are known | | Disable a feature | Flag behavior is understood, including in-flight work and users who depend on it | | Pause writes or workers | Backlog, expiry and downstream consequences are acceptable and observed | | Shift traffic | Destination has capacity, correct configuration and required state | | Apply a forward fix | Scope is bounded, test evidence is adequate for the risk, and a recovery path exists | | Repair data | Affected set is known, backup or audit evidence is preserved, and business reconciliation is approved |

During suspected compromise, involve security before changes that could destroy evidence or continue unsafe access. Preserve relevant logs and state within the organization's access and retention controls. Incident command is not permission to bypass every security boundary.

Run separate investigations in parallel where useful, but coordinate production changes. When feasible, observe the effect of one change before adding another. If urgent containment requires multiple actions, record their order and owners so the resulting state is still explainable.

Worked incident: duplicate export attempts

This fictional exercise concerns a billing export worker, not an Ampity customer. A new retry path can resend an export after a timeout even when the receiver accepted the original. The team has a tested way to pause the worker without losing queued items.

| Time | Evidence or decision | | --- | --- | | 14:00 | A release introduces the faulty retry behavior; this is later confirmed as impact start | | 14:04 | A duplicate-attempt signal alerts the team | | 14:06 | The responder acknowledges and begins checking scope | | 14:08 | An incident commander is named; application and finance contacts join | | 14:12 | The affected worker is paused; queue growth and deadlines are monitored | | 14:20 | The team confirms an older worker is compatible with current schema and queued messages | | 14:35 | The compatible worker resumes safe new processing; uncertain prior exports remain quarantined | | 15:10 | Reconciliation identifies and resolves the affected exports; the incident is ready to close |

The pause limits further duplicate attempts but also delays exports. That tradeoff is accepted explicitly. Rolling back the worker does not correct prior external submissions, so reconciliation remains a separate obligation.

The receiver's records and stable export IDs determine which uncertain items were already accepted. The team does not blindly replay the quarantined set. Items that cannot be verified remain held for an authorized decision.

Detection took 4 minutes from the confirmed impact start. Acknowledgement took 2 minutes after the alert. Initial containment occurred 12 minutes after impact began, safe new processing resumed after 35 minutes, and reconciliation completed after 70 minutes.

These are durations for one scenario, not a mean recovery statistic. Reporting only the 35-minute service restoration would hide the remaining correctness work. If the impact start were uncertain, the record should show that uncertainty rather than choose a convenient timestamp.

Communicate facts and the next decision point

A useful update separates confirmed impact from investigation. Avoid promising a recovery time before the team has evidence.

For the exercise, an update could read:

This is a fictional example, not a statement about an active incident. Use the organization's approved customer communication channels and review process. Do not publish sensitive customer records, exploit details or speculative blame.

Set the next update time based on the incident's needs and available staff. If nothing material has changed, say what remains unknown and what the team is checking. Avoid making the technical lead answer the same questions in several channels while also executing recovery.

Define recovery before declaring it

A green infrastructure dashboard is evidence, not the acceptance test by itself. Identify the customer-visible conditions that must hold.

Check that the critical journey works, error and latency signals have stabilized under relevant traffic, queued work is draining safely, and any correctness issues have an owner. Confirm that temporary mitigation has not disabled an essential security or durability control.

Some incidents can close with a clearly tracked repair task; others cannot safely close until repair is complete. State the reason and obtain the required business or security acceptance. Keep temporary flags, capacity changes and access grants on a removal register.

If recovery depends on a workaround, record its limitations and what would trigger escalation again. Do not let “resolved” conceal an unowned operating risk.

Convert the review into tested changes

Use the incident record to examine why the system and response behaved as they did. Separate the triggering event from contributing conditions such as missing idempotency, incomplete tests, ambiguous ownership or delayed access.

For each follow-up, record an owner, priority, verification method and due date appropriate to its risk. “Improve monitoring” is not a testable action. “Alert when accepted export IDs repeat, and exercise the alert with a synthetic duplicate” is specific enough to verify.

Review whether actions were completed and whether the same failure condition remains possible. Count incidents and recovery durations with clear definitions, and inspect distributions and impact rather than drawing conclusions from one average.

Practice the playbook with a tabletop or controlled exercise. Test an unavailable primary responder, a failed paging integration and a recovery path that cannot be used. An exercise should expose missing authority and dependencies before a customer incident does.

Keep the tooling and service boundary current

Select incident tooling for the workflows you need: escalation, schedules, audit history, integrations and a fallback communication path. Verify its lifecycle and migration requirements.

As checked in September 2026, Atlassian's Opsgenie lifecycle notice states that new sales ended on June 4, 2025 and support ends on April 5, 2027. It should not be treated as an unqualified new-tool recommendation. Existing users need to test their migration, including schedules, integrations, permissions and alert delivery, before retiring the old path.

For a practical next step, choose one critical service and fill its role roster, mitigation gates, customer recovery test and communication template. Run the exercise with the people who would actually respond.

Ampity's DevOps and SRE service is a relevant route for discussing those operational controls. Support hours, escalation duties and response commitments must be agreed for the engagement; this guide does not imply an included 24/7 managed-response service.