Jenny’s food blog
Jenny started a new food blog. There will be much food porn there.
Jenny started a new food blog. There will be much food porn there.
After seeing this essay about shaving make the rounds for the 3rd or 4th time, I decided that I should actually try it. So, this morning I tried shaving with nothing more than hot water and a fresh blade. Much to my amazement, it seems to have worked just fine. Who knew? I’ll see what happens if I attempt that method several times in a row. I fully expect my skin to get more irritated than normal.
… so of course they pick that time to break. My backups server at home (wakko), which is a mac mini, decided to melt its internal hard drive on about Dec 31. Sadly it took me until Jan 4 to notice, because I don’t have ironclad enterprise-grade monitoring of my home network. (Of course, what I do for a living is maintain an enterprise-grade monitoring system. Irony!)
It took about 30 minutes for the recovery process to turn into this scenario, as such things always do. Internal disk in mini dies. Of course there’s no serial port on the mini, nor is there a monitor in the closet where it lives, so I take it out and attach it to the tv in the bedroom, where I discover that there’s rampant filesystem corruption, which isn’t fixable because of I/O errors on the internal disk. OK, call it a loss - let’s attach its external disks (which are in USB+Firewire enclosures) to an older Thinkpad that conveneintly also has Linux installed. Wait, what? What are all these USB timeout errors? These things worked perfectly with the mini! Oh, it turns out these enclosures work great when attached via Firewire, and not so well via USB. Actually it turns out one of them is fine, but the other one own’t work at all via USB. Oh wait, the fine one isn’t so fine - its fan has siezed up and the disk is about to catch fire. Thankfully, I have a spare disk enclosure to replace that one. Still only one working disk out of two required to get my backup pool mounted, though. Hmmm… aha, I have another spare disk enclosure, but it’s the one I don’t like because it traps heat. Let’s try it anyway. Yay, it works! Now to get LVM started. WTF, device mapper not working? Hm … manually load the kernel modules. Try again. Woo, lvm works now. Filesystem mounted!
Now what? Um … I dunno. I guess I should get backuppc installed on this new machine so backups start working again. At least I didn’t actually lose data to this. In theory even if the backup pool was lost I wouldn’t be losing data, becuase hey, it’s backups!
I think the lesson here is that even my backups server wants to have redundant storage so I don’t have to waste a day fixing it every time it throws a disk.
Oh, and some monitoring stuff wouldn’t hurt. What do people use for monitoring their stuff? Nagios seems to be miserable - is there something nicer out there that’s free?
Comments(1)