A lot of things seem obvious when the perfect once in a lifetime storm hits. But it’s tough to consider doomsday when you’re doing cost analysis for data centers or network operations centers.
Power Generation – It appears that some hosting / web infrastructure providers located power generators in areas which can flood. Here’s a quote from a NY Times Article “When Flood Waters Rise, Web Sites May Fall”
As more of life moves online, damage to critical Internet systems affect more of the economy, and disasters like Hurricane Sandy reveal vulnerabilities from the sometimes ad hoc organization of computer networks. In places like Manhattan, advanced technology comes up against aging infrastructure and space constraints, forcing servers and generators to use whatever space is available.
It seems the Google and Amazon Web Services were unaffected by this storm. By sheer geographical diversity and making mistakes in the past, these behemoths have figured out redundancy and disaster planning better than most.
Verizon lost quite a bit of wired telephone service (and I’m assuming DSL because of that.) We advise our clients to consider a wireless data / Internet redundancy option and according to reports, there was far less outage in wireless vs. wired. We strongly believe that wireless will be the first to come after a major event as well. Here’s how they prepped:
In the days leading to the hurricane, the carriers staged fleets of emergency response vehicles — trucks that act as temporary cell towers — in strategic locations along the storm’s edge. They also took safety measures, like installing backup batteries on cell sites and moving important equipment to less vulnerable areas. They advised customers to use text messaging instead of placing phone calls to use fewer network resources.
The article goes on to discuss some folks who had planned ahead with a generator on high ground. In their words,
“We’ve got a truck full of diesel pulled up to the building, and now we’re trying to figure out how to get fuel up to the 19th floor.”
Planning for disaster is no fun and isn’t cheap. But if you plan on making it through a major disaster with inconvenience as your major discomfort, get help developing a plan to either evacuate your key hardware, software, and documentation in time to ride out the storm safely. Or, move key operations into the cloud well ahead of a problem. Some disasters like tornado, fire, theft, and earthquake don’t give us days of warning and advanced computer model tracking. They strike and we react.
Prayers go out to all those affected by this amazing act of nature.