The Failed Militia
The hum of the Zoom bridge is a physical weight, a low-frequency vibration that settles right behind my eyes where the bruise from the glass door is still throbbing. There are 31 people on this call. I can see the list of names scrolling on my second monitor, a digital roll call of expensive talent currently doing absolutely nothing. The ‘Incident Commander’-a title that sounds remarkably like a rank in a failed militia-is asking for the fourth time in 11 minutes if there is an update on the database migration. His voice has that practiced, artificial calm you see in flight attendants right before the oxygen masks drop. It’s a performance. We are all performers here. In my day job as a bankruptcy attorney, I see this same theatricality when a board of directors realizes the cash flow has hit zero. They don’t reach for a solution; they reach for a procedure manual.
I walked into a glass door this morning. It was one of those floor-to-ceiling sheets of architectural hubris that are so clean they cease to exist as matter. I was looking at my phone, checking a notification about a 501 error, and then-thud. My forehead met the physical manifestation of transparency.
❝
That is exactly what your incident response process is: a beautifully polished surface that everyone swears is a path forward until they actually try to move through it. We’ve adopted the language of the elite-the scribes, the commanders, the communications leads-but we’ve forgotten that these roles were designed to reduce cognitive load, not to create a bureaucracy that demands 301 percent more energy than the actual fix.
The Private Reality
In the main Slack channel, the ‘Scribe’ is dutifully logging every time someone says ‘we are looking into it.’ It is a chronological record of our collective ignorance. Meanwhile, in a private DM that contains exactly 11 engineers, the actual work is happening. They are sharing raw logs, swearing at legacy code, and ignoring the main bridge entirely. They have to. The main bridge is too noisy, filled with stakeholders who want to know the ‘Estimated Time of Recovery’ every 21 minutes so they can update a status page that no one reads until the service has been down for 41 minutes anyway. This bifurcation of the incident-the public performance and the private reality-is where the real danger lives. It creates a shadow infrastructure of knowledge that disappears the moment the bridge closes.
Incident Noise vs. Fix Activity
I once handled a bankruptcy for a firm that had 201 separate policies for ‘Ethical Asset Disposal’ but didn’t have a single person who knew where the physical keys to the warehouse were. We are doing the same thing in DevOps. We buy the books, we attend the workshops, and we appoint a ‘Communications Lead’ whose only job is to translate ‘we are screwed’ into ‘we are investigating a localized degradation of service.’ It’s a linguistic lie that buys us nothing. When the pressure is on, we don’t rise to the level of our roles; we fall to the level of our least-practiced habits. And our habit, universally, is panic. But now, it’s panic with a spreadsheet.
The process is the scar tissue of a wound that never properly healed.
The Comfort of Labels
I’m sitting here with an ice pack on my face, watching the ‘Incident Commander’ interrupt a Senior Engineer who was actually starting to explain the deadlock. The Commander wants to know if this is a ‘Priority 1’ or a ‘Priority 1-Alpha.’ This is a distinction that exists only in the mind of someone who has never had to actually roll back a deployment at 3:01 AM. We love these labels because they feel like control. If we can name the beast, we imagine we can tame it. But the beast doesn’t care about your taxonomy. The database is still locked, the users are still getting 501s, and I am still wondering why I didn’t see that glass door.
The Legal Trap
There is a specific kind of arrogance in assuming that a process can replace intuition. In the legal world, we call this ‘procedural due process’-as long as we followed the steps, the outcome is technically valid, even if it’s a disaster. In tech, we’ve fetishized the ‘Blameless Post-Mortem’ to the point where it’s become a ‘Fact-less Post-Mortem.’ We are so afraid of hurt feelings that we refuse to acknowledge that some people are simply bad at handling pressure. We put them in ‘Commander’ roles because they are organized, forgetting that an incident isn’t a project to be managed; it’s a fire to be extinguished. You don’t ask a project manager to lead a crew of smokejumpers.
We need to talk about the ‘Scribe’ role. In theory, the Scribe is there to capture the timeline for the post-mortem. In practice, the Scribe is a court stenographer for a trial that has no judge. They capture the ‘what’ but never the ‘why’ because the ‘why’ is happening in those 11 private DMs I mentioned earlier. If your incident response doesn’t capture the frantic, messy, and often incorrect guesses that lead to the fix, your post-mortem is just a piece of fiction. It’s a sanitized version of history that makes us feel safe enough to go back to sleep. I’ve read thousands of pages of bankruptcy filings, and they all read the same: ‘Market conditions shifted.’ They never say ‘The CEO spent $500001 on a yacht while the payroll account was empty.’
Honesty in the Heat
This is why I find myself gravitating toward the raw, unpolished insights found in
Ship It Weekly, where the reality of shipping code isn’t hidden behind a layer of corporate-speak. We need more of that honesty in the heat of the moment. We need to be able to say, ‘I don’t know what’s happening, and having 31 of you watching me is making it harder to find out.’ But we don’t. We maintain the facade. We keep the bridge open for 121 minutes of silence, punctuated by the Commander asking for a ‘status check.’ It’s a waste of human potential that makes my legal heart ache. Each of those engineers costs the company roughly $201 an hour. Do the math. We are burning a mid-sized sedan’s worth of capital every time we have a poorly managed ‘managed’ incident.
$42,000+
(Based on 20 engineers at $201/hr)
Building for Ghosts
I think back to the glass door. The problem wasn’t the door itself; it was the lack of visual cues. There was no ‘Push’ sign, no sticker at eye level, no smudge of a previous victim’s forehead. Our incident processes are often too transparent. We see the roles, we see the Slack channels, we see the Jira tickets, but we don’t see the actual constraints. We don’t see the fact that the person in charge is terrified of the CTO, or that the primary engineer hasn’t slept in 21 hours. We build our processes for the ‘Ideal Engineer’ in a ‘Vacuum,’ a creature that doesn’t exist. Real engineers are tired, they are grumpy, and they are currently talking about you behind your back in a private Discord server.
Documentation is not a substitute for a conversation.
The Fluid Hat
We should also stop treating the ‘Incident Commander’ as a permanent role. It should be a hat that gets passed to whoever has the clearest view of the problem at that moment. Sometimes that’s a junior dev who happens to be the only one who touched the CSS today. Sometimes it’s the person who’s been at the company for 11 years and remembers the ‘Great Outage of 2011.’ When we rigidify these roles, we prevent the natural emergence of leadership. We force a hierarchy onto a situation that requires a network.
The True Cost Check
Speed Over Steps
Prune participants aggressively.
Capture the Why
Document the failures, not just the steps taken.
Fluid Command
Pass the hat to the person with the clearest view.