What time is it? Redux

So, you're having problems with the Macosx bundled NTP and want to install your own? I've heard from a few people who have gotten to this point and are looking for a little help. So, here's a really quick and dirty set of instructions to download, compile, and install the latest production version of NTP on your Mac. These instructions assume you have already installed Xcode from the Mac App Store.

Open the Terminal App and execute these commands one at a time. Choose a starting directory in your home directory. You also need to be logged into an admin account so have the right privileges to install into /usr/local/bin at the end.

curl -o ntp-4.2.6p5.tar.gz http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-4.2/ntp-4.2.6p5.tar.gz
gunzip ntp-4.2.6p5.tar.gz 
tar xf ntp-4.2.6p5.tar 
cd ntp-4.2.6p5
sudo make install
ls -ltr /usr/local/bin/ntp*

The output of the ls command should be look something like this:

-rwxr-xr-x1 rootwheel622488 Feb8 16:53 /usr/local/bin/ntpd
-rwxr-xr-x1 rootwheel270660 Feb8 16:53 /usr/local/bin/ntpsnmpd
-rwxr-xr-x1 rootwheel266636 Feb8 16:53 /usr/local/bin/ntpq
-rwxr-xr-x1 rootwheel266080 Feb8 16:53 /usr/local/bin/ntpdc
-rwxr-xr-x1 rootwheel124372 Feb8 16:53 /usr/local/bin/ntpdate
-rwxr-xr-x1 rootwheel171640 Feb8 16:53 /usr/local/bin/ntp-keygen
-rwxr-xr-x1 rootwheel2029 Feb 27 17:23 /usr/local/bin/ntptrace
-rwxr-xr-x1 rootwheel1394 Feb 27 17:23 /usr/local/bin/ntp-wait

Hope this helps.



What Time is It?

Why time is important

Remember in old days of DOS and you had to set the date and time on the with the date and time commands every once in a while? Just like a cheap watch, the clock chip really wasn't very accurate so you were trained to accept the need to correct the clock periodically. In the mid-90's or so, you first connected to the Internet and computers could synchronize clocks across the network. Remember the old rdate command in Unix? Yeah, good times.

Long before then, D. L. Mills wrote RFC-958, "Network Time Protocol (NTP)", and started what has become the de facto standard technique for synchronizing computers across networks. It's become so prevalent that even Windows includes an NTP client! (Of course with Windows, you configure it with some crazy registry settings instead of sensible config files.) NTP has matured to the point where most computer users don't think twice about the computer's clock because it just works.

But, why do we care how accurate the computer's clock truly is? We're not measuring particle physics experiments here, I'm just reading emails! Well, maybe one or more of these reasons might be something you care about:

  1. So the kids can’t use the computer’s clock as excuse for not: showering, feeding the pets, getting homework done, going to bed.
  2. Some network applications are sensitive to gross time differences between computer clocks.
  3. It’s 2014 and the clock on the VCR is still flashing - Blinking twelve problem. We have the technology to do better than this.

Symptoms on Mac OS X

Out of the box

I think we all expect modern operating systems to arrive with a reasonably good default configuration that gets you up and running quickly. Fresh out of the box, your Apple computer is a pretty quick study and only asks a few basic questions before it's connected to the Net. Computer clock synchronization is straightforward and Mac OS X does have reasonable default settings. Your Mac running Mavericks synchronizes with a region-specific Apple Network Time Protocol server. In my case, that is time.apple.com which is a fine NTP server. Note that the time.apple.com FQDN represents a pool of NTP servers. Execute “dig +short time.apple.com” to see evidence of that. Your Mac boots up, starts the NTP process, and your computer’s clock synchronizes without any work on your part. No more blinking twelves.

Unfortunately, your Mac will not remain in sync for long. While setting up my older MacMini as an Open Directory replica, I noticed that the clock was many seconds out of sync with respect to the OD Master. While I the clock sync was causing problems, it rubbed me the wrong way that my computers couldn't perform the equivalent of walking and chewing gum at the same time.    

I opened the Date & Time Preference Pane and verified that "Set date and time automatically" was enabled and configured to use time.apple.com.

This should configure NTP correctly, but it doesn't work correctly.

This should configure NTP correctly, but it doesn't work correctly.

I toggled the checkbox off then on to give NTP a little kick in the pants. After a few seconds, the clock shifted and was in sync with my other computers. Easy-peasy, right? Well, not quite.

After a few days, I noticed that the clock on this particular machine was once-again many seconds out of sync. At this point, I mentally replayed the Seinfeld episode in which Jerry reserves a car, but there's not a car to be found…

”Mac OSX knows how to set the clock sync, Mac OSX just don’t know how to maintain clock sync and that’s really the most important part, the maintaining.”

“sntp time.apple.com” uncovered that my computers were inconsistent with one another and they were not synchronized with time.apple.com. At this point, I could see how this was going to play out: Easy Peasy was not an option anymore, it was time for a deep dive into the innards of computer clock synchronization.

Now, you might look at the length of this article and think "It can't be that broken!" Trust me, I attempted to find "low impact" solutions many times based on all the existing online discussions I could find. None of them truly solved the clock sync problem for me. If you find a viable solution without requiring this level of effort, by all means, please let me know.

Network Time Protocol

Let the computer figure out what time it 

The Network Time Protocol (NTP), RFC-5905, describes a technique allowing computers on the Internet to synchronize their clocks to one another. NTP defines a hierarchical NTP server structure (kind of like pyramid) composed of different strata. The higher stratum (lower numeric value) nodes are topologically closer to a high quality source of time (like a GPS receiver or maybe even an atomic clock). An individual higher stratum NTP node at stratum N will service NTP requests from a number of nodes at stratum N+1, and those stratum N+1 nodes will in turn each service requests from many stratum N+1 nodes, thus distributing the high quality time source to many lower stratum clients (like the computer in front of you), without having 30 million MacBook Pros pounding on the few high quality time servers.

NTP works in a client/server fashion, where the client and server quickly fling packets back and forth at each other, meticulously labeling each packet with the local clock time of when it was sent and received. (In reality, NTP can operate in a number of different modes, client/server being just one of them. However, it's the most used mode for home users.) By analyzing these time stamps using some well-specified math, the client can determine statistics about the network path between the client and server and the relative clock offset between the client and server.

Ignoring the complexity of how NTP arrives at this conclusion, let’s suppose the client determines that the network latency is 10 seconds (wow! I hope not!). If the server sent a packet with a server timestamp of 14:00:00.00UTC and the client received it at 14:00:12.00UTC according to the client's clock, then the client will say, “Hey, my clock is two seconds ahead of that NTP. I need to fix that!" That's my two-sentence hand-wavy explanation of NTP. I'll explain a little more of the details as I tell my story.

How many servers should you attempt to peer with? There's a parable: “A man with one watch always knows what time it is, a man with two watches is never quite sure?” I’ve seen some terribly misconfigured time servers which pop up in public NTP pools periodically: Don’t set yourself up with a single point of failure. A well-configured NTP client will attempt the synchronization exchange with a number of servers. Four or five is probably a pretty good number. The client will interrogate each time server, evaluate all of the results, and choose one of the servers as the one to trust. Again, the exhaustive detail is in the specification of NTP.

Now that the NTP client has peered with a good server, what should it do? Well, one tenet of time is that time monotonically increasing and doesn’t go backwards (at least not until we invent the flux capacitor, anyway). Going back to the example synchronization from above, NTP can’t just throw the computer’s clock in reverse back to 14:00:10.00UTC! That would mean that your computer would relive those two seconds in some alternate universe...not a good idea. Similarly, large forward jumps in time would also be disruptive and are also to be avoided. NTP needs to make small but frequent adjustments to the clock to avoid disruptive clock jumps.

Over long periods of time (minutes to hours), an NTP client continues to poll the servers and  determines how many “clock ticks” the computer’s clock is drifting (in terms of parts-per-million) and writes the drift value into a file called ntp.drift. That value is different for every computer, and will even vary over time for any individual computer. Once NTP has a good drift estimate, NTP wakes up every second and makes tiny adjustments to the internal clock frequency of the computer to gradually bring it into sync and keep it there. Whew! And that’s just the glossy brochure version of NTP.

The Network Time Protocol functionality has been implemented primarily in the imaginatively named ntpd daemon and a few associated utility programs. (For information about ntpd and all things NTP, start at the NTP Project home or you can cut to the chase and go to the Network Time Synchronization Research Project.) While ntpd can utilize a number of configuration files, the interesting one is /etc/ntp.conf. Typically, that file lists the time servers that the client will interrogate. The typical M.O. for managing the life-cycle of ntpd is straightforward and relatively standard on a unix-ish system:    

  1. Configure /etc/ntp.conf with some number of servers    
  2. During boot time, the OS will do a few things via a one or more startup scripts:    
    1. Execute sntp to make very gross time corrections early in the boot process    
    2. Start ntpd with the proper command line options    
    3. Ensure that ntpd will restart if it fails (it’s a daemon, after all)    

The Mac OS X NTP Environment

Mac OSX has modified the NTP Daemon

The ntpd lifecycle is controlled by the /usr/libexec/ntpd-wrapper script. Nothing odd here since most unix operating systems have a script to manage the lifecycle of most daemon processes. More about how this works later. Unfortunately, the Mavericks ntpd-wrapper script has some problems.

  • Mac OSX 10.9.1 version supported only the simplest of server configurations    
    • No support for “pool” entries, only “server” entries    
    • No support for server association options, such as the highly recommended “iburst” option. Using such features would actually cause errors at startup.    
  • Mac OSX 10.9.2 fixed those limitations in /usr/libexec/ntpd-wrapper.

Mavericks introduced pacemaker into the NTP ecosystem. pacemaker is a Mac OSX daemon that attempts to reduce the impact of clock synchronization on battery life. Pacemaker allows fine-grained control over the frequency with which the clock will be adjusted. Apple introduced this layer of control under the assumption that frequent updates to the computer clock will circumvent the OS’s attempts to keep the CPU idle as much as possible to save battery power. However, the ntpd community believes ntpd has little impact on power consumption and pacemaker addresses a non-issue. Pacemaker uses different adjustment intervals depending on whether running on AC or battery power. See “man pacemaker” or do a little searching on the web. By itself, the concept of pacemaker doesn't seem like it should make a huge different on clock synchronization, however, Apple actually moved the responsibility for adjusting the clock completely into pacemaker and removed that functionality from the MacOSX 10.9.x ntpd distribution. This has turned out to be problematic.

Mac OSX 10.9.x has a modified NTP Daemon

The Mac OSX 10.9.x ntpd will query servers as normal, derive the computer clock’s drift, update the ntp.drift file. This actually appears to work, at least for some short period of time after ntpd is restarted. However, Mac OSX ntpd does not adjust the computer’s clock at all, because pacemaker now performs that job. Also, after a while the Mavericks ntpd simply starts rejecting all NTP servers.

Date & Time Preference Pane

  • Leads you to believe a single time server is sufficient. You can enter multiple time servers in that Preference Pane by comma-separating the values, but that doesn't allow you to include any NTP options.
  • The Preference Pane clobbers any customizations made to /etc/ntp.conf. Fortunately, you can configure NTP to your own specs quite well without ever using the Date & Time Preference Pane, so this clobbering can be avoided by ignoring the Preference Pane.

What’s wrong with NTP in Mavericks?

There's quite a bit of history on NTP problems in Mac OSX. I only noticed issues beginning with the Mac OSX 10.9.1, so I'm not considering what might have been happening in the pre-Mavericks days. If you want to read up on the reported problems, here are links to some of the more recent and relevant discussions:

There are also quite a few relevant discussions on Apple's Support forums. I've read most of them, I've tried many of them, and while some of them initially resolve clock sync issues, they don't address all of the problems that I've found.

Deep Dive on inspecting the problem

If your Mac has been running for more than a few days and you compare your Mavericks clock with a better source of time, e.g. Apple's Time Server, you’ll probably notice that your clock is different. To really quantify the difference, run this command in a Terminal window:

cygnus:~ user$ sntp time.windows.com
2014 Mar 01 19:29:53.845391 +1.1 +/- 0.046295 secs

That shows a +1.1 second offset from Apple’s time server on an unmodified 10.9.2 iMac that has been running for just 3 days. Give it a few more days and that offset will continue to grow: My 10.9.1 systems were 10's of seconds out of sync. Here’s an example of the same command on a “repaired” 10.9.2 MacBook Pro.

orion:~ user$ sntp time.apple.com
2014 Mar 02 16:59:24.9871 +0.005949 +/- 0.005661 secs

In this case, the offset is less than 6 milliseconds...much better. Still not convinced? Want to see some other debug output from an unrepaired 10.9.2 system?

cygnus:~ user$ ntpq -c pe -c as
     remote           refid      st t when poll reach   delay   offset  jitter
 x-plane.waitone      3 u  826   64    3    0.282   25.263  43.154    2 u  763   64    1   69.603   41.587  34.588
 tock2.usshc.com .GPS.            1 u  698   64    1  109.476    6.248  16.566
 falcon.ca.us.sl    2 u  891   64    1   86.285   35.751  41.774
 xen1.rack911.co    3 u  889   64    1   84.967   38.992  38.525

ind assid status  conf reach auth condition  last_event cnt
  1 34353  9014   yes   yes  none    reject   reachable  1
  2 34354  9034   yes   yes  none    reject   reachable  3
  3 34355  9014   yes   yes  none    reject   reachable  1
  4 34356  9014   yes   yes  none    reject   reachable  1
  5 34357  9034   yes   yes  none    reject   reachable  3

That output uses ntpq ("ntp query") to show ntpd’s current state for each of the associated servers. All but the first server in the list are from the 0.us.pool.ntp.org NTP server pool. What’s worth noting is that ntpd has rejected every potential NTP server. The Apple-modified ntpd has given up and isn’t peering with any servers. (The observant user will notice that I did modify /etc/ntp.conf to include additional servers beyond the default.) Here’s the same command from a repaired 10.9.2 installation:

orion:~ user$ ntpq -4c pe -c as
     remote           refid      st t when poll reach   delay   offset  jitter
-watt.22pf.org  2 u  391 1024  377   22.236    8.664   2.516
+ns1.technotic.c  3 u  943 1024  377   18.286    6.770   3.498
*vcxen02.bw.phl.   2 u   63 1024  377   15.712    7.468   0.797
+ntp1.housing.be    2 u 1971 1024  376   93.100    6.204   2.429
-mail.honeycomb.    3 u  280 1024  377   42.294    5.763   7.698
-ccadmin.cycores   2 u  191 1024  377   29.587    4.147   2.553
-sisdb01.muskego  3 u  731 1024  377   47.663   10.910   6.139
-fush.slash31.co  2 u   29 1024  377   85.637   11.146   1.814
-time.apple.com   2 u  320 1024  377   22.112    2.890   3.932

ind assid status  conf reach auth condition  last_event cnt
  1 57364  933a   yes   yes  none   outlyer    sys_peer  3
  2 57365  9424   yes   yes  none candidate   reachable  2
  3 57366  961a   yes   yes  none  sys.peer    sys_peer  1
  4 57367  943a   yes   yes  none candidate    sys_peer  3
  5 57368  9324   yes   yes  none   outlyer   reachable  2
  6 57369  931a   yes   yes  none   outlyer    sys_peer  1
  7 57370  9324   yes   yes  none   outlyer   reachable  2
  8 57371  931a   yes   yes  none   outlyer    sys_peer  1
  9 57372  931a   yes   yes  none   outlyer    sys_peer  1

That is a sight for sore eyes! There are a few candidates, a number of outlyers (sic), no rejects, and ntpd has found a good peer. Accurate clock sync is achievable in Mavericks!

If you want an accurate clock in Mavericks    

This is not for the feint of heart: These problematic components are part of the OS. I tried a number of different ways to fix the existing components, but have simply not found a way to make it work. This is transplant time, so scrub up and prep for surgery!    

Outline of the work

  1. Shutdown pacemaker - it's getting in the way and we don't need it. ntpd has been optimized over the years to minimize its impact on the host computer. My computers run on AC 99% of the time, so even if pacemaker does help with battery life, it’s not a factor for me.    
  2. Replace the built-in ntpd and associated utilities - Apple's ntpd distribution is hobbled, plain and simple. You can leave the existing executables in place if you’d like, they will just not be used.    
  3. Improve on the default NTP configuration by using NTP pools and just one additional option in /etc/ntp.conf.

Shutdown pacemaker    

We're going to turn off pacemaker, put it on the shelf, and leave it there for safe keeping. We don't want it. Pacemaker is managed by launchd, which in turn is managed by the launchctl command. In the terminology of launchctl, you need to unload the pacemaker configuration.

cygnus:LaunchDaemons user$ cd /System/Library/LaunchDaemons
cygnus:LaunchDaemons user$ sudo launchctl unload com.apple.pacemaker.plist

Pacemaker will never rear its head again, unless you “load” it manually.    

Download, compile, and install Clean NTP Binaries    

OK, we're now going to dive a little deeper, but the NTP distribution is relatively easy to build.

  1. You will need Xcode. It's in the Mac App Store and you've already paid for it by buying an Apple computer.
  2. Download the latest Production version of NTP from the NTP Source Distribution website.
  3. Build and install NTP.
  4. You should wind up with these nice new executables in /usr/local/bin:
orion:~ user$ ls /usr/local/bin
ntp-keygen  ntpd     ntpdc  ntpsnmpd  sntp
ntp-wait    ntpdate  ntpq   ntptrace  tickadj

Modify /usr/libexec/ntpd-wrapper to use the Clean NTP binaries.    

ntpd-wrapper is executed by launchd. During boot up, ntpd-wrapper first executes sntp to make coarse clock adjustments, then starts ntpd which will run forever and make fine adjustments to the system clock. The original sntp and ntpd executables are still loaded on your computer and, unless you modify root’s executable search path, they will be executed by default. We need to ensure that ntpd-wrapper executes the new Clean NTP binaries.

  1. Using your favorite editor (vim is more than up to this task), look for “if sntp” and “exec ntpd” in /usr/libexec/ntpd-wrapper.    
  2. Prepend “/usr/local/bin/“ to sntp and ntpd in /usr/libexec/ntpd-wrapper.

The last 9 lines of ntpd-wrapper should look like this when you're done:

for server in $(awk '/^server/ {print $2}' /etc/ntp.conf); do

if /usr/local/bin/sntp -K /dev/null -s ${server} &> ${LOG}; then
    logger -p daemon.err -f ${LOG}


exec /usr/local/bin/ntpd -c /private/etc/ntp-restrict.conf -n -g -p /var/run/ntpd.pid -f /var/db/ntp.drift `

(Need a screen shot here)

Improve the default /etc/ntp.conf    

You have many options available to you at this point. I personally include time.apple.com in my configuration. The NTP Pool project is a terrific clearinghouse of public NTP servers, so I’ve added that to the mix. I’ve also experimented with using time.nist.gov, but this particular machine isn’t using that right now.

driftfile /var/db/ntp.drift    
pool 0.us.pool.ntp.org iburst    
pool 1.us.pool.ntp.org iburst    
server time.apple.com iburst

This configuration works very nicely.    

Finally, let’s restart using the new ntpd. But before you restart, how do you know whether you’re using old or new versions?

cygnus:bin user$ pgrep -lf ntpd    
116 /usr/sbin/ntpd -c /private/etc/ntp- restrict.conf -n -g -p /var/run/ntpd.pid -f /var/db/ntp.drift

Notice “/usr/sbin/ntpd”, that’s the original ntpd.    

Restart like this:    

cygnus:bin user$ sudo launchctl stop org.ntp.ntpd    
cygnus:bin user$ sudo launchctl start org.ntp.ntpd    
cygnus:bin user$ pgrep -lf ntpd    
11145 /usr/local/bin/ntpd -c /private/etc/ntp-restrict.conf -n -g -p /var/run/ntpd.pid -f /var/db/ntp.drift

Notice "/usr/local/bin/ntpd" and recall that's the new clean version that you just built and installed.

At this point, ntpd will begin building up statistics about each of the potential servers and will eventually decide to peer with one of them. Use the ntpq commands that I demonstrated earlier to inspect the synchronization state. Monitor using ntpq and sntp (or ntpdate) over the next few hours and days to verify that your Mac:

  1. does not reject all (and probably not any of) the time servers
  2. ntpd and your computer's clock gradually improves time sync with respect to the NTP servers.

Don’t expect sub-millisecond accuracy when synchronizing across the Internet like this, but you can expect the clock to be within 10-20 milliseconds most of the time. Three of my four Macs have an estimated offset of less than 10 milliseconds, while the fourth is running totally "open loop" with Apple's ntpd...it's marching to its own beat!

This article seems very long, but it's really scraping the surface of the Network Time Protocol. I originally went through this effort with Mac OSX 10.9.1 and verified that these fixes work (and are still necessary) with Mac OSX 10.9.2. If you're having computer clock issues on your Mavericks Mac, as was I, these steps are the only way I've found to get your computer's clock on straight and narrow. Good luck!



Don't try to outsmart the internets

You know it's bad when your kids give you that look: "Dad, what have you done to the network this time?!". My wife is a saint and takes it all in stride, but the kids haven't yet reached that level of home-networking nirvana.

The family frequently sits down together to watch a few YouTube videos on the AppleTV (3rd gen). It's fun for all of us, the oldsters get to keep up with the internet memes and the kids get to practice their acting skills when looking shocked by some of the videos.

About 4 weeks ago, though, it wasn't a fun experience. There were huge lags before the videos began to play and they would typically pause every 15-20 seconds waiting for more download. Oddly, movies from Apple seemed just fine. Our computers reported 25+ Mbps download speeds. How could this be? Must be some blowback of the great Apple/Google wars, (of course mainly Google's fault, no doubt! :)

I finally sat down and researched problems with AppleTV and YouTube content. There's a lot of noise on this topic going back a few years, but I think most of it was just that...noise. Finally, I came across some informative articles describing DNS issues that can arise when using alternate DNS providers (like OpenDNS) while accessing content that is provided by big Content Delivery Networks (CDNs), think Akamai and Google for example. I had recently switched from my ISP's recommended DNS server to OpenDNS (with a DynDNS backup) so this seemed worth pursuing. I still believe OpenDNS provides a valuable service, but this problem might be a sticking point for some situations.

I ran namebench to test my DNS settings and possibly make some other recommendations. To my surprise, my ISP's DNS server responded faster than any of the other DNS servers that namebench tested, including OpenDNS and DynDNS. Seemed promising, but would this resolve the AppleTV/YouTube problems.

I ran the Mountain Lion Server App to adjust the DNS server settings and also change the appropriate DHCP settings (both of these services are provided by Mountain Lion Server, not by my ISP's router...). Zip, boom, voila! Back to super-fast YouTube videos on the AppleTV.




The World of Disks Will Fall.

As I am want to do, I'll adapt the fine works of Tolkien:

A day may come when the Logical Block Addressing fails, when we forsake our Sector Management and break all bonds of Volume Integrity."

Yes, it's happened again. For (only) the second time in my computer life I've lost a hard drive. (OK, third if you include the time I dropped a laptop, but that was just stupidity!). This time it was particularly annoying because, a) it's the system drive on my Mac Mini Lion Server and b) I was about to finish all of my FreeNAS and iSCSI machinations. Yes, I was about to leave the role of System Administrator which had been consuming me for the better part of five months and become a productive User again. Oh well.

The insidious thing about this failure is that it blind-sided me with really strange symptoms that took a while to piece together. The system began getting progressively slower but nothing was obviously wrong: memory was fine, no high CPU-load, lots of free disk space, and no errors in the logs. I first went to Disk Utility and ran Repair Permissions which corrected some problems, but it didn't resolve the performance issues. I then started investigating with BlackMagic Disk Speed Test (it's free and I recommend you get it now from the Mac App Store). Write performance on the system drive was fine, but read performance was in the gutter. But Disk Utility thought the disk was fine...what gives?

I then tried resetting PRAM (now I'm getting really desperate), but that didn't help. I finally tried the old free version of SMARTReporter which, after running the two hour test, reported disk read errors. Finally, some evidence of a problem.

Great, now what? I have Time Machine, but since the drive was just mostly dead, I decided to clone it to an external drive. Break out trusty Carbon Copy Cloner (I really need to send them some money one of these days) and let it rip. It was slow because the source drive was failing, but I wound up with a bootable clone of the bad drive. This is huge since the Mac Mini is my Lion Server for our household network...my kids and I are in the dark without this (well, except for iPhones and iPads).

I set up a Genius Bar appointment and Chris, The Genius, confirmed the problem and ordered a replacement drive. I decided to take the Mac Mini home and continue using it with the external drive. The repair will be done in-store but they'll probably need it for a day or two since he said there's some non-trivial surgery involved in replacing that drive. The retail cost of this whole exercise is about $200, but the real cost is $0 with the $99 AppleCare Protection Plan that I purchased. Money well spent!

 This whole event event has made me review my backup strategies. I think I'm in pretty good shape, but I'm definitely not following the rule of three. The Mac Mini system drive is definitely a single point of failure. I may need to set up a nightly clone of the system drive. The life of a home system administrator never ends.



iSCSI - sensible or obsession?

In previous posts I've alluded to my ongoing search for a workable network storage solution for a Mac family. I don't think it's asking too much to sit down at Mac in the house to get at your 200GB of home folder goodness. Apparently, it's harder than I thought.

Right now, it's about an 80% solution with a mashup of a Mac Mini running Lion Server for Directory Services and a Freenas server in the basement serving up the home folders using the Apple Filing Protocol (AFP) using netatalk. My last post had a workaround for one particular application not playing well with AFP, but I still have other lingering problems:

  • Neither iMovie nor Final Cut Pro really work with an AFP home folder. Ouch.
  • Lion's Auto-Save and Versions feature doesn't work with AFP. Come on, Apple!
  • Spotlight insists on re-indexing from scratch every time one of us logs in, slowing things down to a crawl.
  • iTunes sometimes says it can't properly save things...maybe it's AFP, maybe it's not, but I'm suspicious at this point.
  • Some apps are just plain slow, even though I'm getting 60+MBps throughput to my storage. See my prior blog about iBank was behaving before I switched my iBank data file a sparse HFS disk image.

Finally, only my two kids and I are rolling with an AFP home folder: No way I can switch my wife over to this setup at this point, so she's "stuck" with a traditional account only accessible from the 27" iMac (could be worse, I know). Can you imagine living in a house with 3 Macs and only being able to access your home folder from one of them. What kind of world is that!

So, what to try now? Freenas with the Zetabyte File System (ZFS) gives me highly reliable storage for all our stuff. I believe the problem lies with how that storage is made available: AFP doesn't play nice (even with Apple's own apps). The two other traditional network-based file systems, NFS and CIFS, aren't any better I suspect. The common theme with all of these is that they are file-based network storage and they simply don't "smell right" to the applications. Applications want to see a nice standard HFS+ file system, but that's not what they get. How can we give 'em what they want?

iSCSI is a network storage protocol that, instead of serving up files, serves up raw blocks of storage that the client side (the Mac) can partition using HFS+ just like a physical disk. OK so far. The applications should see a nice HFS+ file system that they're expecting, but the data is actually stored on my robust Freenas ZFS-based storage. Other than the fact that the complexity has just gone through the roof and we're now moving into enterprise-level storage technologies, what could possibly go wrong?

We'll see.



Don't forget about Disk Utility

I was one of the many who abandoned Quicken on the Mac when MacOSX Lion came out. Apple had been continuing to support older, end-of-life, PowerPC Macs through a number of major OS releases and they finally just stopped when Lion came out. I have no qualms with Apple on this: Most software vendors with a good business plan were on board with Universal releases of their Apps within months...with the notable exception of Intuit. Intuit just couldn't seem to get with the program in time, forcing users like me to decide between Quicken and the next version of MacOSX.

Good bye, Quicken. Hello, iBank. iBank was different, but did everything I needed and they were producing new releases on a regular basis.

Fast forward to my latest network setup with home folders served up from a central Network Attached Storage (NAS) device (more on that some other day). Now, instead of zippy performance, iBank was slowed to a crawl when doing the simplest data entry tasks. How could this be? After all, my NAS actually was faster than Firewire drives on some of my machines.

After a conversation with the nice support people at IGG Software, turns out they flat out don't support anything but data files stored on a local drive. No network storage (even using Apple File Protocol). No cloud storage.

So, what can you do? Turn to your old friend, Disk Utility, Swiss-army knife of disk and file system magic. This gem has been around in MacOSX forever and never gets the respect it deserves. Among its many tricks, it can create disk images. What's a disk image you ask? Think of it like a self-contained hard drive packed into what looks like a regular old file. MacOSX can mount this disk image and that mounted image now behaves just like a real file system. So, if iBank wants to think it's using a plain old disk, why not try to fool it into believing it's true. Let's create a small disk image, store the iBank data file in it, and see if iBank has better performance.

 Use spotlight to find and start "Disk Utility". When it starts, you should see information about the hard drive(s) accessible from your computer. Click on the "New Image" button along the top of the window and enter some details about the disk image you'd like to create. Choose a size that makes sense for your needs. Here's an example screenshot:

Don't forget to give your new disk image a "Save As" name. I used "storage". Then click "Create". Excellent! You now have a new disk image and Disk Utility should have automatically mounted it for you. Disk Utility should look a little different now:

Now, just double click on the "My Disk Image" icon on your desktop to open it in Finder, drag your iBank data file into that folder, and open that file in iBank. Performance problem, solved! And we've really pulled a fast one on iBank using the power of disk images, because the data is really still stored on the Network Drive.

I bet you can do this same trick and store the storage.dmg file somewhere in the cloud and get the same performance benefit. If you try that, I urge you to try out something other than "none" for the Encryption option shown in the first screen shot. Better safe than sorry.

Have fun. Time to update my iBank records!



There must be a way to make this work!

Household chores are never really fun, but you do them so you can eat dinner using clean plates and your clothes don't smell like the dog: Chores are a means to an end. In the old days, you learned how to take care of all the things in the house from your parents when you were growing up and that knowledge transferred across generations. For example, my washing machine is a few evolutionary generations beyond what we had growing up, but everything I learned in the 70's to operate a washing machine mostly applies today: put the dirty stuff in, add some detergent, hit a couple buttons, and pull out the clean clothing 45 minutes later. On a daily basis, we use many of the same appliances that we used 30-40 years ago and in basically the same way, except for one: Yep, you guessed it, computers.

As much as we might want to, we can't treat computers as simple appliances. The set of tasks we accomplish with computers changes all the time; the kids are doing school projects, we're sharing songs and pictures with everyone in the family, playing games, sending/receiving email, managing all of our calendars, doing finance tasks (it's just about tax time, right?), and making online purchases. It's tough (impossible?) to make an appliance that adapts to so many different uses. (Maybe the integration of iOS features into MacOSX will improve the situation, but only time will tell at this point.)

To top it all off, we want access to our apps, docs, and web sites from any of the devices in the house. When I get bumped off the computer by the kids, I want to be able to switch to a laptop and pick up with the same docs that I was just working with. Maybe "the cloud" could help but I'm a skeptic and I also find comfort in being able to lay my hands on the device where my data lives. So, let's also demand reliable, shared, and permanent storage from our computing appliances. Oh, and just for fun, maybe you want to keep your private files private, too. This is starting to sound like drudgery.

As much as the great minds in Cupertino and similar places keep trying, none of our computers behave like appliances. They need attention, and sometimes it feels like lots of it. It seems like not a day goes by where the kids aren't asking about setting up some VPN software to support a multi-player online game or the website for the summer camp registration doesn't work with our browser. Imagine if you had to call the Appliance Repair guy to come over every other day to help you press the right buttons on your dishwasher? Seems ludicrous, but that's essentially what it's like with our computers. So, I'm all about finding ways to help push computers more into "appliance-space" and less in "consume-all-your-free-time land."

Now, I'd be lying if I said I didn't enjoy the challenge sometimes. After all, if the solution to a problem is a new gadget or some great new piece of software, who am I to say no to that? Beating down computer problems by applying a little new tech or putting together some sort of "mash-up" solution yourself can be lots of fun.

So, my plan here is to talk about the journey through the home IT scene and hopefully help make some of this problem more like a simple appliance than the insufferable pain that it sometimes can be. I'll talk about some situations I've faced some ideas to work through it. As they say, "Watch this space!"