Teach Us, Oh Great Doomsday Whistle!

Every Friday at 11am sharp the County tests out the Disaster Warning Sirens near World Headquarters here in Indianapolis. You know–the painfully loud sirens that will theoretically warn you moments in advance of the coming tornado (if the green sky and twilight at noon didn’t clue you in) so that you might have time to bend over and kiss your butt good-bye.

Since my desk is conveniently located just one block from the siren, and we rely on the good graces of mother nature to cool the office in Indiana summer, not much gets done during this two minute “disaster drill” every Friday morning because the noise-level is… distracting. At best.

Despite enduring this diligent preparation, week-in and week-out for 12 of the last 15 years (that I’ve lived in Indiana) I have yet to experience one actual natural disaster. My eyes have not as yet drunk in the sight of a big fat F-5 tornado, for example, coming to get its “$5 Footlong” before smashing the Subway sandwich shoppe across the street to rubble–possibly in protest of its failure to honor the traditional “ducks eat free!” policy.

So why do I bring this up? Even though it is irritating and disruptive, I am glad that they test the alert system regularly. The last thing I need is to get caught flat-footed in the event of nuclear meltdown, tornado, or a Godzilla/Mothra attack–these sorts of drills prepare everyone and ensure all of our long-term safety. Here at headquarters, we take civil-defense seriously: It is a vital component of disaster recovery for the city.

And just as the city prepares for its potential calamities with regular drills, so should you with your Information Technology Assets.

Data-Disaster Drills

When was the last time you RESTORED your backup? Do you know its usable? Can you demonstrate this to your own satisfaction for data critical to your enterprise? A backup is only as good as it is reliable: If what you’re capturing can’t be used to restore your systems to the desired state, they’re pretty much useless.

At a minimum, you should conduct disaster drills in your test environment with your backups from the live environment. Can you access mailboxes on the Exchange server? Do back-end applications like databases or AD services (DNS/DHCP/File/Print) contain needed objects? Take inventory and audit against the live environment. At the end, have a designated test-user attempt to access vital systems in test to verify resource availability. Troubleshoot all problems you encounter during this test and resolve them, so your next test (or actual disaster) goes better than your test(s).

How Often to Blow Your Own Siren?

How often (and how loudly) will vary widely depending on what your business does, what industry you’re in, whether that business is heavily regulated, and your own tolerance for risk. A small shop with a small-revenue stream will approach this question differently than a multi-billion dollar multi-national that is regulated by a Federal agency like the SEC or FDA.

With that in mind, I would suggest you make your schedule like this:

Do it often enough that you’re confident you can use your plan and your backup to save the company (and your job) in an emergency. In a simple enterprise you can probably achieve this level of comfort doing a drill once or twice per year. Larger organizations will conduct quarterly drills on less-critical systems, monthly on important ones, possibly even weekly drills on specific functions of a recovery scheme for enterprise-critical functions.

The idea is this: In a small shop with 4-5 servers you can easily do a restore to Virtual-machines and a “functionality verification” in a few hours twice annually without breaking the bank or being negligent.