On March 22 & 23, 2018 the Blackrock 3 Partners is hosting a 2-day open enrollment training session on IT Incident Management and Incident Command, designed for SRE personnel and others with the responsibility to be involved with incident resolution. Go to our home page www.blackrock3.com for more information about the class. This article is an excerpt from our textbook titled Incident Management for Operations by O’Reilly Media, which will be provided to the participants of the class.
The Blackrock 3 Partners have translated the Incident Management System (IMS) which is used by the emergency services in the United States for use by SRE personnel. One of the main components of this system is to identify an Incident Commander. Without leadership and a process to organize and direct personnel and resources, the effort to solve a problem may end up being as chaotic as the problem itself. Therefore, for any incident to be successfully managed, there must be one person in charge – the Incident Commander. Here are some thoughts on the role of the Incident Commander (IC).
I’m the Incident Commander – Now What?
At this point in your SRE career, you may or may not have had the opportunity to lead an incident. If you have had the pleasure, you know it can be a stressful and overwhelming endeavor. If you haven’t, there will likely be a day in the future when you’ll be called upon to lead a group of people during an incident. Leading an incident is challenging and mentally taxing, especially in those situations where the solution to the problem isn’t obvious, the pressure is on, and time is ticking away.
Some people may criticize your decisions while others praise your leadership. Your actions may be quietly, or, in some cases, not so quietly, questioned or second-guessed. After the event is over and it becomes clear what caused the problem, some may question why it took so long to resolve. Of course, if you had all the answers at the beginning of an event that you have at the end, you may have made different decis ions.
Ultimately you will earn some sort of reputation as an Incident Commander (IC). It may be good or bad, but it will certainly follow as well as precede you to every incident response you lead. It is therefore critical to your team and yourself to start off on the right foot and adopt a few key principles.
First and foremost, when you are in charge of an incident, you are an Incident Commander. Your job is to lead, and lead you must. You may facilitate and encourage discussion, but it is done from your viewpoint as an Incident Commander, using a wide variety of interpersonal skills (active listening, time management, etc.) to direct the effort. You may use the IMS as a framework to organize the personnel, but IMS is a tool for management, not a substitute for leadership. Back in the introduction we offered a simple distinction between management and leadership, but it bears repeating here – things are managed while people are led.
An IC must have command presence – the ability to establish and assert yourself as the leader on the incident bridge. Someone who has command presence has the personality, voice, tone, and poise that alerts everyone on the bridge that he/she is in command. The IC sets the tone for the incident, and when the IC is calm, cool, and collected, it will set the atmosphere for other responders. An IC that can’t make a decision, or changes decisions after they’re made, will eventually lose control of the incident. Once the IC loses control, it is very difficult to get back, and causes the IC to waste valuable time trying to regain control.
As the IC, you will make many decisions. These range from deciding initial strategy and tactics with Subject Matter Experts (SME) input, to officially terminating the incident response and everything else in between. There is also a fine balance between having command presence and being a difficult, overbearing leader. For some ICs, their style is to lead by intimidation, which does not foster open, honest communication. Good ICs are assertive, but also know how to identify and work with a variety of personality traits to achieve maximum results.
Many new ICs become overwhelmed with all the decisions that need to be made and fall into some common traps, from having an opinion on everything (micromanaging), to letting the incident solve itself by allowing the group to hash out a solution on an uncertain timeline (schedule creep). Additionally, a common mistake ICs make is to take the initial position of IC, but end up getting dragged into the weeds of solving the problem, thereby losing the big picture perspective. This doesn’t mean you abandon your skill set, knowledge, or experience. It is just a caution to avoid getting pulled into the detailed problem solving instead of commanding. Take your skill set to the position of command – it may certainly help you investigate or diagnose the problem, or identify a proposed solution. ICs must be in command, leading a group of problem solvers, but not solely engaged in solving the problem themselves. To avoid this, the IC should focus on the process of command, even if he or she is an expert in the problem area.
When done right, the IC should be so focused with their role that they have no bandwidth to get into the fine details of solving the problem. To that end, here are a few FAQs of incident command:
1. I am an Incident Commander. I also have experience and expertise as a database administrator (database SME) and the problem is clearly within our database. Should I work on solving this database problem myself or be the Incident Commander for a response team of SMEs?
ANSWER: You don’t have to be a technical expert in the problem area to be the Incident Commander. It may be helpful to have a working knowledge of the affected systems, provided you don’t lose perspective and start engineering instead of commanding, but it’s not required. The Incident Commander must be an expert in IMS, while the SME are experts in their respective technical domains.
2. I am the Incident Commander. Do I have to make all the decisions?
ANSWER: The Incident Commander is ultimately responsible for all the decisions made but does not have to make all of them. A good IC will create an environment of lively and open discussion amongst the right mix of technical expertise. The goal is to draw information out of the SME (and yourself) to understand the problem and formulate a plan of attack to solve the problem. The IC should not be a heavy-handed dictator that directs each and every move on the conference bridge. Rather, the IC is there to extract every bit of information from the SMEs, form it in to a useful plan, keep the response focused and moving forward, and declare an end point when the incident is resolved.
3. I am a senior executive within the organization during peacetime. Should I always be the Incident Commander?
ANSWER: No. Incident command is a function not a specific person and the position can be filled by anyone qualified to fill it. When an incident occurs, the incident signals to the entire organization the shift to wartime and therefore the peacetime organization chart may not apply. The IC needs to be excellent at leading the response, and that does not always correlate to seniority or rank within the Peacetime organization chart. For example, a junior person with good command presence may be a more logical choice to lead an incident over someone more senior with better technical skills, but who may be introverted. The owner of the baseball team doesn’t come onto the field to pinch hit in the bottom of the ninth inning when the game is on the line. That’s why the players were hired – to play the game. The owner’s job revolves around managing the business of the game. The players are responsible for playing it at all times and in all circumstances.
The IC must be aware of what is currently going on with the incident but also must be continually thinking about the future. They must be several steps ahead of the SMEs on the incident bridge. They must be tracking the SMEs progress and status of the incident while trying to determine the next few steps. It is usually a struggle to develop a plan A to resolve an issue, but the IC must be also develop (or at least be thinking about) plan B and possibly multiple other alternative plans (C, D, E…). As with other responsibilities, the IC is responsible for these alternative plans, but does not have to develop these plans alone. The planning function can be delegated to a separate team on a separate bridge and report their progress to the IC. For example, if the SMEs have offered a solution to solve the problem, and the IC approves their plan, the next words from the IC should be, “let’s discuss alternative plans (plan B),” even while the SMEs are implementing plan A.
It is the role of the IC to set the strategic tone for the incident and leave the individual tactics to the SMEs. In many cases, the IC has the team debate the pros and cons of a solution – but the IC makes the final call.
Come to the seminar and learn more about the process of leading and managing people during an incident. Since you have read this far use the discount code: FOFLN We’ve been doing it for decades in the fire service and IT critical infrastructure and our clients have seen dramatic improvement in their times to resolve! Here’s a quote from a recent attendee of one of our trainings.
“I have worked in the IT industry for over 20 years and was not sure what to expect from this course or how it could help me. Within 48 hours of completing the course, we began applying some of the concepts learned and have already noticed a difference in the flow of information as well as how the calls are managed. We have already heard from some of our leadership about the improvements they have noticed, specifically in the cadence of the call and the flow of information. This course is exactly what we needed. I look forward to being an advocate for this process in my organization.”
The Blackrock 3 Partners have translated the Incident Management System (IMS) which is used by the emergency services in the United States for use by SRE personnel. One of the main components of this system is to identify an Incident Commander. Without leadership and a process to organize and direct personnel and resources, the effort to solve a problem may end up being as chaotic as the problem itself. Therefore, for any incident to be successfully managed, there must be one person in charge – the Incident Commander. Here are some thoughts on the role of the Incident Commander (IC).
I’m the Incident Commander – Now What?
At this point in your SRE career, you may or may not have had the opportunity to lead an incident. If you have had the pleasure, you know it can be a stressful and overwhelming endeavor. If you haven’t, there will likely be a day in the future when you’ll be called upon to lead a group of people during an incident. Leading an incident is challenging and mentally taxing, especially in those situations where the solution to the problem isn’t obvious, the pressure is on, and time is ticking away.
Some people may criticize your decisions while others praise your leadership. Your actions may be quietly, or, in some cases, not so quietly, questioned or second-guessed. After the event is over and it becomes clear what caused the problem, some may question why it took so long to resolve. Of course, if you had all the answers at the beginning of an event that you have at the end, you may have made different decis ions.
Ultimately you will earn some sort of reputation as an Incident Commander (IC). It may be good or bad, but it will certainly follow as well as precede you to every incident response you lead. It is therefore critical to your team and yourself to start off on the right foot and adopt a few key principles.
First and foremost, when you are in charge of an incident, you are an Incident Commander. Your job is to lead, and lead you must. You may facilitate and encourage discussion, but it is done from your viewpoint as an Incident Commander, using a wide variety of interpersonal skills (active listening, time management, etc.) to direct the effort. You may use the IMS as a framework to organize the personnel, but IMS is a tool for management, not a substitute for leadership. Back in the introduction we offered a simple distinction between management and leadership, but it bears repeating here – things are managed while people are led.
An IC must have command presence – the ability to establish and assert yourself as the leader on the incident bridge. Someone who has command presence has the personality, voice, tone, and poise that alerts everyone on the bridge that he/she is in command. The IC sets the tone for the incident, and when the IC is calm, cool, and collected, it will set the atmosphere for other responders. An IC that can’t make a decision, or changes decisions after they’re made, will eventually lose control of the incident. Once the IC loses control, it is very difficult to get back, and causes the IC to waste valuable time trying to regain control.
As the IC, you will make many decisions. These range from deciding initial strategy and tactics with Subject Matter Experts (SME) input, to officially terminating the incident response and everything else in between. There is also a fine balance between having command presence and being a difficult, overbearing leader. For some ICs, their style is to lead by intimidation, which does not foster open, honest communication. Good ICs are assertive, but also know how to identify and work with a variety of personality traits to achieve maximum results.
Many new ICs become overwhelmed with all the decisions that need to be made and fall into some common traps, from having an opinion on everything (micromanaging), to letting the incident solve itself by allowing the group to hash out a solution on an uncertain timeline (schedule creep). Additionally, a common mistake ICs make is to take the initial position of IC, but end up getting dragged into the weeds of solving the problem, thereby losing the big picture perspective. This doesn’t mean you abandon your skill set, knowledge, or experience. It is just a caution to avoid getting pulled into the detailed problem solving instead of commanding. Take your skill set to the position of command – it may certainly help you investigate or diagnose the problem, or identify a proposed solution. ICs must be in command, leading a group of problem solvers, but not solely engaged in solving the problem themselves. To avoid this, the IC should focus on the process of command, even if he or she is an expert in the problem area.
When done right, the IC should be so focused with their role that they have no bandwidth to get into the fine details of solving the problem. To that end, here are a few FAQs of incident command:
1. I am an Incident Commander. I also have experience and expertise as a database administrator (database SME) and the problem is clearly within our database. Should I work on solving this database problem myself or be the Incident Commander for a response team of SMEs?
ANSWER: You don’t have to be a technical expert in the problem area to be the Incident Commander. It may be helpful to have a working knowledge of the affected systems, provided you don’t lose perspective and start engineering instead of commanding, but it’s not required. The Incident Commander must be an expert in IMS, while the SME are experts in their respective technical domains.
2. I am the Incident Commander. Do I have to make all the decisions?
ANSWER: The Incident Commander is ultimately responsible for all the decisions made but does not have to make all of them. A good IC will create an environment of lively and open discussion amongst the right mix of technical expertise. The goal is to draw information out of the SME (and yourself) to understand the problem and formulate a plan of attack to solve the problem. The IC should not be a heavy-handed dictator that directs each and every move on the conference bridge. Rather, the IC is there to extract every bit of information from the SMEs, form it in to a useful plan, keep the response focused and moving forward, and declare an end point when the incident is resolved.
3. I am a senior executive within the organization during peacetime. Should I always be the Incident Commander?
ANSWER: No. Incident command is a function not a specific person and the position can be filled by anyone qualified to fill it. When an incident occurs, the incident signals to the entire organization the shift to wartime and therefore the peacetime organization chart may not apply. The IC needs to be excellent at leading the response, and that does not always correlate to seniority or rank within the Peacetime organization chart. For example, a junior person with good command presence may be a more logical choice to lead an incident over someone more senior with better technical skills, but who may be introverted. The owner of the baseball team doesn’t come onto the field to pinch hit in the bottom of the ninth inning when the game is on the line. That’s why the players were hired – to play the game. The owner’s job revolves around managing the business of the game. The players are responsible for playing it at all times and in all circumstances.
The IC must be aware of what is currently going on with the incident but also must be continually thinking about the future. They must be several steps ahead of the SMEs on the incident bridge. They must be tracking the SMEs progress and status of the incident while trying to determine the next few steps. It is usually a struggle to develop a plan A to resolve an issue, but the IC must be also develop (or at least be thinking about) plan B and possibly multiple other alternative plans (C, D, E…). As with other responsibilities, the IC is responsible for these alternative plans, but does not have to develop these plans alone. The planning function can be delegated to a separate team on a separate bridge and report their progress to the IC. For example, if the SMEs have offered a solution to solve the problem, and the IC approves their plan, the next words from the IC should be, “let’s discuss alternative plans (plan B),” even while the SMEs are implementing plan A.
It is the role of the IC to set the strategic tone for the incident and leave the individual tactics to the SMEs. In many cases, the IC has the team debate the pros and cons of a solution – but the IC makes the final call.
Come to the seminar and learn more about the process of leading and managing people during an incident. Since you have read this far use the discount code: FOFLN We’ve been doing it for decades in the fire service and IT critical infrastructure and our clients have seen dramatic improvement in their times to resolve! Here’s a quote from a recent attendee of one of our trainings.
“I have worked in the IT industry for over 20 years and was not sure what to expect from this course or how it could help me. Within 48 hours of completing the course, we began applying some of the concepts learned and have already noticed a difference in the flow of information as well as how the calls are managed. We have already heard from some of our leadership about the improvements they have noticed, specifically in the cadence of the call and the flow of information. This course is exactly what we needed. I look forward to being an advocate for this process in my organization.”
RSS Feed
