As I am want to do, I'll adapt the fine works of Tolkien:
A day may come when the Logical Block Addressing fails, when we forsake our Sector Management and break all bonds of Volume Integrity."
Yes, it's happened again. For (only) the second time in my computer life I've lost a hard drive. (OK, third if you include the time I dropped a laptop, but that was just stupidity!). This time it was particularly annoying because, a) it's the system drive on my Mac Mini Lion Server and b) I was about to finish all of my FreeNAS and iSCSI machinations. Yes, I was about to leave the role of System Administrator which had been consuming me for the better part of five months and become a productive User again. Oh well.
The insidious thing about this failure is that it blind-sided me with really strange symptoms that took a while to piece together. The system began getting progressively slower but nothing was obviously wrong: memory was fine, no high CPU-load, lots of free disk space, and no errors in the logs. I first went to Disk Utility and ran Repair Permissions which corrected some problems, but it didn't resolve the performance issues. I then started investigating with BlackMagic Disk Speed Test (it's free and I recommend you get it now from the Mac App Store). Write performance on the system drive was fine, but read performance was in the gutter. But Disk Utility thought the disk was fine...what gives?
I then tried resetting PRAM (now I'm getting really desperate), but that didn't help. I finally tried the old free version of SMARTReporter which, after running the two hour test, reported disk read errors. Finally, some evidence of a problem.
Great, now what? I have Time Machine, but since the drive was just mostly dead, I decided to clone it to an external drive. Break out trusty Carbon Copy Cloner (I really need to send them some money one of these days) and let it rip. It was slow because the source drive was failing, but I wound up with a bootable clone of the bad drive. This is huge since the Mac Mini is my Lion Server for our household network...my kids and I are in the dark without this (well, except for iPhones and iPads).
I set up a Genius Bar appointment and Chris, The Genius, confirmed the problem and ordered a replacement drive. I decided to take the Mac Mini home and continue using it with the external drive. The repair will be done in-store but they'll probably need it for a day or two since he said there's some non-trivial surgery involved in replacing that drive. The retail cost of this whole exercise is about $200, but the real cost is $0 with the $99 AppleCare Protection Plan that I purchased. Money well spent!
This whole event event has made me review my backup strategies. I think I'm in pretty good shape, but I'm definitely not following the rule of three. The Mac Mini system drive is definitely a single point of failure. I may need to set up a nightly clone of the system drive. The life of a home system administrator never ends.