Portolan Operations decision desk
Open Portolan
All playbooks
Playbook · Process & quality

Incident response and on-call

Incident load is produced by the system: noisy alerts, unclear ownership, change-induced failures and a rotation that is too small. This path measures the load, fixes routing and command before adding people, and protects rest as a design constraint.

Start a decision

6 steps · 4 cited sources · free during beta

The question this path answers

How should we handle incidents and on-call so the service recovers quickly without burning out the team?

The path, step by step

  1. Measure the incident system before changing it

    For four weeks, record volume by severity, time to acknowledge, time to the owning team, reassignment count, pages per shift and rest-day pages. Segment by service and change type.

    Why it worksThe 20-system study found between 4.11% and 91.58% of reports were reassigned and misrouting added up to 10.16 times the triage time. Google's on-call practice treats pages per shift and hours per incident as load limits, not as individual performance.

    EvidenceChen et al. (2019) · Beyer et al. (2016)

    CheckpointA baseline table with median and worst-case values for each measure, by service.

    Common failureDebating whether to add a person using anecdotes instead of a measured load and routing baseline.

  2. Fix ownership and escalation before adding responders

    For the categories with the most reassignment, name a single owning team and a first-response rule. Test the rule against the last 50 incidents and tighten ambiguous categories.

    Why it worksIncorrect assignment is frequent and costly, especially for high-severity incidents, and automated triage helps only partly. Explicit ownership and escalation rules are the part the team can control now.

    EvidenceChen et al. (2019)

    CheckpointEvery common incident category has one owner, and a backtest shows fewer reassignments than the baseline.

    Common failureBuying a triage tool or adding a second on-call before defining who owns what.

  3. Separate command, operations and communication

    Declare an incident when it spans teams, is customer-visible, or remains unresolved after an hour. Assign incident commander, operations lead, communications and planning roles, and write them in a live incident document.

    Why it worksGoogle's incident management practice separates responsibilities so one responder is not debugging, communicating and coordinating at once. A clear handoff keeps command continuous when the shift changes.

    EvidenceBeyer et al. (2016)

    CheckpointMajor incidents have named roles and an explicit handoff; no single person holds all roles.

    Common failureLetting the on-call engineer do command, debugging and stakeholder updates simultaneously.

  4. Cut paging load at the source

    Rank alert rules by the pages they cause and the action they produced. Remove or downgrade alerts that need no action, fix the top recurring causes, and set a ceiling on pages per shift.

    Why it worksGoogle caps on-call at 25% of an engineer's time and treats more than about two incidents per 12-hour shift as unsustainable. Change-induced incidents are a large share, so change and alert hygiene reduce load before staffing does.

    EvidenceBeyer et al. (2016) · Zhao et al. (2023)

    CheckpointPages per shift fall, and the share of alerts that lead to no action falls with them.

    Common failureAdding a second rotation while the same noisy alerts keep firing.

  5. Protect rest and rotation

    Cap on-call frequency, protect the rest day, define swap rules and track rest-day pages. If the team cannot staff a sustainable rotation, change scope or adopt a follow-the-sun rotation before stretching the same group further.

    Why it worksThe on-call health review associates on-call work with stress, sleep disturbance and safety risk, and finds the evidence thinner than for shift work. Google's rotation math shows a 24/7 primary-plus-secondary rotation needs at least eight engineers on one site at a 25% cap.

    EvidenceNicol & Botterill (2004) · Beyer et al. (2016)

    CheckpointZero rest-day pages, on-call within the agreed cap, and no shift above the incident limit.

    Common failureTreating burnout as a staffing surprise after people resign.

  6. Review the life cycle and close the loop

    For each major incident, write a blameless postmortem that links the change, timeline, detection, assignment and mitigation stages. Assign actions with owners and review repeat causes monthly.

    Why it worksThe 231-incident study links incidents to the change that introduced them and to the stage where resolution stalled. Postmortem and follow-up work are part of the six-hour incident estimate, not an optional extra.

    EvidenceZhao et al. (2023) · Beyer et al. (2016) · Chen et al. (2019)

    CheckpointA postmortem within five days, actions closed by the agreed date, and a falling repeat-incident rate.

    Common failurePublishing a blame narrative that stops people reporting the next incident.

The evidence base

Each step above cites hand-selected research. These are the studies behind the path.

More from the library

Further reading from Portolan's research library, matched by subject. These papers are not proof that any step works; check each checkpoint against your own operation.

  • REASON (1995) — Understanding adverse events: human factors.
  • Tucker (2004) — The impact of operational failures on hospital nurses and their patients
  • Sharma et al. (2005) — Systematic failure mode effect analysis (FMEA) using fuzzy linguistic modelling
  • Wu et al. (2017) — An Evidential Reasoning‐Based CREAM to Human Reliability Analysis in Maritime Accident Process
  • Su et al. (2015) — Dependence Assessment in Human Reliability Analysis Using Evidence Theory and AHP
  • Qiu et al. (2024) — Failure risk management: adaptive performance control and mission abort decisions
  • McFadden et al. (2009) — The patient safety chain: Transformational leadership's effect on patient safety culture, initiatives, and outcomes
  • Dabhilkar et al. (2016) — Supply-side resilience as practice bundles: a critical incident study
  • Paté‐Cornell (1993) — Learning from the Piper Alpha Accident: A Postmortem Analysis of Technical and Organizational Factors
  • Nateghi et al. (2011) — Comparison and Validation of Statistical Methods for Predicting Power Outage Durations in the Event of Hurricanes

Other playbooks