Sometimes, there is an edge case we didn’t anticipate that causes issues in production.
And sometimes, there is a common use case we didn’t test sufficiently that causes issues in production.
And sometimes, production goes down. For reasons yet unknown.
For some of us, there is a production support team that handles these things. They might call us, if they need us, but they handle this stuff. For the rest of us, we need to handle the incident ourselves.
The following is a very simple framework for responding to incidents. It is a brief write-up on roles and key steps for resolving incidents. This primarily serves as a guideline for situations such as production outages or incidents that impact multiple users and adversely affect the ability to do business. This is a starting place. It is lightweight but for many organizations it is all the formality they’ll ever actually need.
Roles
There are four key roles; Command, Recorder, Comms, and Research
Each of these roles needs to be handled. One person can handle multiple roles, but it is best to have at least one individual per role whenever possible. Most roles can have multiple people, if staffing allows or the scope is large enough to warrant a larger group. Only one person can be in the command role at a time.
The fifth role, support, is optional but very valuable in larger scale longer-term incidents.
Command - Runs the Room
This role is coordinating all efforts.
Command does not necessarily make decisions, but can do so if there is a stalemate or time does not allow.
Command is responsible for managing the approach and focus.
Command is the voice most commonly heard in the room.
No action that potentially impacts our ability to do business is to be taken without express confirmation from command
For example
Code deployments
Data updates
Things command is managing:
Overall Approach
Focus
Short term (now)
Long term (in hours)
Identify who needs to be involved
Know who is working on what
Recorder - Takes the Notes
This role is recording incoming data in a way that is accessible to all participants.
Recorder needs to pay attention
People may not explicitly share the data with recorder
Things recorder is managing:
Record incoming information
Make information visible to all who need it
Answer command’s questions
Comms - Shares the Status
This role is managing communication; primarily external and secondarily internal.
Status updates to stakeholders on a regular cadence
Brief Update
Incident Name (include incident ID if one exists)
Change in status since last update
ETA to resolution
TBD
Days or Hours
Complete
Things comms is managing:
Understand approach and current focus
Send out regular status updates
Contact anyone we need who is not yet involved
Research - Finds the Clues
Research needs to ensure Command and Recorder have information as soon as possible
Things research is managing:
Gather the evidence we need
Run queries
Look through code
Support - Helps the Team
This role is available for support - they may switch into any other role as needed.
Keep room organized
Runner - go get people and things needed
Handle food/drink
Fill in for other roles as needed
Steps
Identification
Identify the specific issue.
If there are multiple issues, make note and determine which one to address first.
Limit WIP - Avoid the temptation to run multiple issues at the same time.
Containment
Take steps to contain the issue - how do we stop it from having further impact?
Disable systems; pause operations?
Communicate to customers?
Impact Analysis
What is the scope of the issue?
How many users are involved?
What communication is required?
Root Cause Analysis
What is causing the issue?
Can we verify the cause?
Can we duplicate the issue in a lower environment?
Mitigation
What are steps to correct the issue?
Do we need to update code or systems?
Do we need to update any data?
Retrospect
How did we do?
What about the process worked?
What about the process didn’t work?
What do we want to do differently in the future?
What about the issue?
What did we learn?
What could we have done to prevent this?