Collaborative Incident Response

This is an idea I've had in my head for a while now, and in light of the recent DDoS attack against Estonia, I got to thinking about it again: the need for collaborative incident response and investigation. The attack against Estonia (which was significant enough to attract the attention of NATO) was effective and performed by individuals who were at least somewhat more sophisticated than your average script kiddie. Before going any further, let me provide a quick background on the idea of collaborative incident response.

Several years ago, I was in charge of designing, training, and implementing a Computer Security Incident Response Team (hereafter referred to as CSIRT) at one of the local hospitals, at which I was employed at the time. The team was well organized and broken into complimentary (and slightly overlapping) areas of expertise. Once everyone on the team was trained and familiar not only with their role but the roles of the rest of the team members, we began performing firedrills in earnest. Scenarios were devised that the CSIRT would address, first as simply roundtable exercises, and then finally real-time, live drills. The idea of the drills was not only to hone the skills of the team, but to identify areas of weakness that we would then attempt to address before the next drill (or actual incident). All told, the drills were effective, useful, and to be perfectly honest, fun. It was at this point that I was contact by our company's disaster preparedness person who told me that there was actually going to be a city-wide disaster drill, and wanted to know if the CSIRT wanted to be included. Naturally, I said yes. I was given the basic scenario for the city-wide drill (a plane crash at the local airport), and then I devised the CSIRT drill around that. Thinking on a city-wide scale really got the ol' wheels turning. What would we do in the event of a security event massive enough to exceed the resources and abilities of the CSIRT? If this was a city-wide (or larger) event, other organizations would potentially be in the same boat, and perhaps we would be able to assist each other.

The ideal situation would be to have local businesses and other organizations come together in a community CSIRT that could be called upon in the event of a significant security incident. Obviously I'm not talking about giving people from other organizations the keys to the kingdom. Far from it. What I am suggesting is having the community CSIRT function primarily in a research and logistical support capacity. In addition, in most cases it would be possible to do this without divulging too much about your inner workings. Let's consider an example. For the sake of our discussion, let's say that our organization, Company Q, is hit with a massive security incident. Key servers are unreliable, portions of our network infrastructure are up and down, workstations all over the enterprise are crashing in a cascading fashion. An event of this size would push the security of any organization to (and quite probably past) the breaking point. Here's where the community CSIRT could come into play. Folks from our organization would be able to sit down with the community CSIRT and begin to dissect the problem. We're in the heat of the battle, so having the assistance of some people who aren't directly affected could be very useful. The initial Crisis Action Meeting would consist not only of our people but key people from the community CSIRT. This would be particularly helpful in identifying the problem as different people from different organizations will bring with them their own experience which, by definition, will be unique. They'll be able to look at the problem in ways that we might not be able to. And if this is a large enough problem, what about fatigue? On the CSIRT that I put together, we implemented the rule that during an incident, a CSIRT member could only put in 10 to 12 consecutive hours before being required to stand down and get some rest. Don't ever underestimate the impact of fatigue during a prolonged engagement. Having some extra people who could be doing things like parsing logs and the like will allow our people to focus on mitigation and recovery, and, as needed, get some rest.

To be certain, there are a number of key points that would need to be worked out in advance. The idea is that it would be a mutually beneficial relationship for all involved. Company Q has an incident so they engage the community CSIRT which consists of Company R and Company S. A month from now, maybe Company R is the one having the problem, and we (Company Q) and Company S lend a hand. It works sort of like the way villages used to fight fires back in the old days; everyone came out to help, because the next house to catch on fire could be yours. Even if your competitors are part of the community CSIRT, they can still be valuable resources.

Obviously, the members of the CSIRT would have to hold themselves to an extraordinarily high ethical standard. Members would have to be chosen carefully. A good place to start my be the local InfraGard chapter. I am the Vice President of my local chapter, InfraGard Springfield. An advantage to starting with InfraGard is that each InfraGard member is vetted by the FBI. Not to say that only InfraGard members could be on the CSIRT, but it is as good of a place to start as any. With some effort, a community CSIRT could become the de facto hub for local IT security matters.