But How Do the Machines Know?

My machines must talk to each other, because the note on my calendar program that I was leaving town incited them to rebel. I left Monday morning for the Macworld Expo in San Francisco knowing that a huge snow storm was coming Tuesday. Sure enough, it hit, melted a bit, became icy, and weighed down power lines. The power was out to the office in which my mail server and some Web sites (including this) operate for a few hours Wed. a.m. But the power came back and everything righted itself thanks to journaling, BIOS settings, and other features.But that evening I discovered that something was wrong with the mail, DNS, and Web server -- it was acting bizarrely and wouldn't respond normally. The only choice was a remote power cycle using my Sophisticated Circuits' PowerKey 650. Works like a charm. Except it didn't. The combination of power cycling damaged some of the boot blocks, which I couldn't determine remotely. An officemate was willing to head in at 11 pm to mess with it. We determined the problem, and I figured out how to fix it (sort of), but then we discovered the keyboard port on that computer was dead. I had an inkling but hadn't put the pieces together a few weeks ago. So I had him start a restore operation from the backup tapes I have. I got ultra-paranoid about backups a few years ago, and have spent more money on backup software and hardware than on practically any other single aspect of my computing life. It paid off. I have an incremental backup that runs every four hours during the day and another that runs at night. I had a 6 pm snapshot of the machine that had died and was able to restore it to another Linux box, very similar in nature. I had already set up secondary onsite and tertiary offsite DNS, and had been moving various sites and resources to a co-located set of servers I'm running that have robust power and bandwidth. The backup took a while, so I went to sleep, got up early, and was able to get email back on line for myself and various others, and make the second machine more or less like the first for general purposes. Later in the day, I was able to get another officemate to swap tapes to restore the Web sites and other features. I was flying back on Thursday night, and was waiting for my plane at 7 pm with an 8 pm boarding time. I found a power outlet and opened my laptop: sure enough, T-Mobile had extended Wi-Fi (and free for the moment) to that terminal. They've slowly been unwiring the whole airport, starting with the easy one, the international terminal, which was built with Ethernet in all the walls. I'm amazed at how quickly I got back to status quo ante and how well Linux and the various GNU and free and open source components cope with being copied, restored, repointed, and so forth. I'm just glad to be back with so few hours invested to where I was. I've called this my most successful complete computer failure ever.