Why time is important

Remember in old days of DOS and you had to set the date and time on the with the date and time commands every once in a while? Just like a cheap watch, the clock chip really wasn't very accurate so you were trained to accept the need to correct the clock periodically. In the mid-90's or so, you first connected to the Internet and computers could synchronize clocks across the network. Remember the old rdate command in Unix? Yeah, good times.

Long before then, D. L. Mills wrote RFC-958, "Network Time Protocol (NTP)", and started what has become the de facto standard technique for synchronizing computers across networks. It's become so prevalent that even Windows includes an NTP client! (Of course with Windows, you configure it with some crazy registry settings instead of sensible config files.) NTP has matured to the point where most computer users don't think twice about the computer's clock because it just works.

But, why do we care how accurate the computer's clock truly is? We're not measuring particle physics experiments here, I'm just reading emails! Well, maybe one or more of these reasons might be something you care about:

  1. So the kids can’t use the computer’s clock as excuse for not: showering, feeding the pets, getting homework done, going to bed.
  2. Some network applications are sensitive to gross time differences between computer clocks.
  3. It’s 2014 and the clock on the VCR is still flashing - Blinking twelve problem. We have the technology to do better than this.

Symptoms on Mac OS X

Out of the box

I think we all expect modern operating systems to arrive with a reasonably good default configuration that gets you up and running quickly. Fresh out of the box, your Apple computer is a pretty quick study and only asks a few basic questions before it's connected to the Net. Computer clock synchronization is straightforward and Mac OS X does have reasonable default settings. Your Mac running Mavericks synchronizes with a region-specific Apple Network Time Protocol server. In my case, that is time.apple.com which is a fine NTP server. Note that the time.apple.com FQDN represents a pool of NTP servers. Execute “dig +short time.apple.com” to see evidence of that. Your Mac boots up, starts the NTP process, and your computer’s clock synchronizes without any work on your part. No more blinking twelves.

Unfortunately, your Mac will not remain in sync for long. While setting up my older MacMini as an Open Directory replica, I noticed that the clock was many seconds out of sync with respect to the OD Master. While I the clock sync was causing problems, it rubbed me the wrong way that my computers couldn't perform the equivalent of walking and chewing gum at the same time.    

I opened the Date & Time Preference Pane and verified that "Set date and time automatically" was enabled and configured to use time.apple.com.

This should configure NTP correctly, but it doesn't work correctly.

This should configure NTP correctly, but it doesn't work correctly.

I toggled the checkbox off then on to give NTP a little kick in the pants. After a few seconds, the clock shifted and was in sync with my other computers. Easy-peasy, right? Well, not quite.

After a few days, I noticed that the clock on this particular machine was once-again many seconds out of sync. At this point, I mentally replayed the Seinfeld episode in which Jerry reserves a car, but there's not a car to be found…

”Mac OSX knows how to set the clock sync, Mac OSX just don’t know how to maintain clock sync and that’s really the most important part, the maintaining.”

“sntp time.apple.com” uncovered that my computers were inconsistent with one another and they were not synchronized with time.apple.com. At this point, I could see how this was going to play out: Easy Peasy was not an option anymore, it was time for a deep dive into the innards of computer clock synchronization.

Now, you might look at the length of this article and think "It can't be that broken!" Trust me, I attempted to find "low impact" solutions many times based on all the existing online discussions I could find. None of them truly solved the clock sync problem for me. If you find a viable solution without requiring this level of effort, by all means, please let me know.

Network Time Protocol

Let the computer figure out what time it 

The Network Time Protocol (NTP), RFC-5905, describes a technique allowing computers on the Internet to synchronize their clocks to one another. NTP defines a hierarchical NTP server structure (kind of like pyramid) composed of different strata. The higher stratum (lower numeric value) nodes are topologically closer to a high quality source of time (like a GPS receiver or maybe even an atomic clock). An individual higher stratum NTP node at stratum N will service NTP requests from a number of nodes at stratum N+1, and those stratum N+1 nodes will in turn each service requests from many stratum N+1 nodes, thus distributing the high quality time source to many lower stratum clients (like the computer in front of you), without having 30 million MacBook Pros pounding on the few high quality time servers.

NTP works in a client/server fashion, where the client and server quickly fling packets back and forth at each other, meticulously labeling each packet with the local clock time of when it was sent and received. (In reality, NTP can operate in a number of different modes, client/server being just one of them. However, it's the most used mode for home users.) By analyzing these time stamps using some well-specified math, the client can determine statistics about the network path between the client and server and the relative clock offset between the client and server.

Ignoring the complexity of how NTP arrives at this conclusion, let’s suppose the client determines that the network latency is 10 seconds (wow! I hope not!). If the server sent a packet with a server timestamp of 14:00:00.00UTC and the client received it at 14:00:12.00UTC according to the client's clock, then the client will say, “Hey, my clock is two seconds ahead of that NTP. I need to fix that!" That's my two-sentence hand-wavy explanation of NTP. I'll explain a little more of the details as I tell my story.

How many servers should you attempt to peer with? There's a parable: “A man with one watch always knows what time it is, a man with two watches is never quite sure?” I’ve seen some terribly misconfigured time servers which pop up in public NTP pools periodically: Don’t set yourself up with a single point of failure. A well-configured NTP client will attempt the synchronization exchange with a number of servers. Four or five is probably a pretty good number. The client will interrogate each time server, evaluate all of the results, and choose one of the servers as the one to trust. Again, the exhaustive detail is in the specification of NTP.

Now that the NTP client has peered with a good server, what should it do? Well, one tenet of time is that time monotonically increasing and doesn’t go backwards (at least not until we invent the flux capacitor, anyway). Going back to the example synchronization from above, NTP can’t just throw the computer’s clock in reverse back to 14:00:10.00UTC! That would mean that your computer would relive those two seconds in some alternate universe...not a good idea. Similarly, large forward jumps in time would also be disruptive and are also to be avoided. NTP needs to make small but frequent adjustments to the clock to avoid disruptive clock jumps.

Over long periods of time (minutes to hours), an NTP client continues to poll the servers and  determines how many “clock ticks” the computer’s clock is drifting (in terms of parts-per-million) and writes the drift value into a file called ntp.drift. That value is different for every computer, and will even vary over time for any individual computer. Once NTP has a good drift estimate, NTP wakes up every second and makes tiny adjustments to the internal clock frequency of the computer to gradually bring it into sync and keep it there. Whew! And that’s just the glossy brochure version of NTP.

The Network Time Protocol functionality has been implemented primarily in the imaginatively named ntpd daemon and a few associated utility programs. (For information about ntpd and all things NTP, start at the NTP Project home or you can cut to the chase and go to the Network Time Synchronization Research Project.) While ntpd can utilize a number of configuration files, the interesting one is /etc/ntp.conf. Typically, that file lists the time servers that the client will interrogate. The typical M.O. for managing the life-cycle of ntpd is straightforward and relatively standard on a unix-ish system:    

  1. Configure /etc/ntp.conf with some number of servers    
  2. During boot time, the OS will do a few things via a one or more startup scripts:    
    1. Execute sntp to make very gross time corrections early in the boot process    
    2. Start ntpd with the proper command line options    
    3. Ensure that ntpd will restart if it fails (it’s a daemon, after all)    

The Mac OS X NTP Environment

Mac OSX has modified the NTP Daemon

The ntpd lifecycle is controlled by the /usr/libexec/ntpd-wrapper script. Nothing odd here since most unix operating systems have a script to manage the lifecycle of most daemon processes. More about how this works later. Unfortunately, the Mavericks ntpd-wrapper script has some problems.

  • Mac OSX 10.9.1 version supported only the simplest of server configurations    
    • No support for “pool” entries, only “server” entries    
    • No support for server association options, such as the highly recommended “iburst” option. Using such features would actually cause errors at startup.    
  • Mac OSX 10.9.2 fixed those limitations in /usr/libexec/ntpd-wrapper.

Mavericks introduced pacemaker into the NTP ecosystem. pacemaker is a Mac OSX daemon that attempts to reduce the impact of clock synchronization on battery life. Pacemaker allows fine-grained control over the frequency with which the clock will be adjusted. Apple introduced this layer of control under the assumption that frequent updates to the computer clock will circumvent the OS’s attempts to keep the CPU idle as much as possible to save battery power. However, the ntpd community believes ntpd has little impact on power consumption and pacemaker addresses a non-issue. Pacemaker uses different adjustment intervals depending on whether running on AC or battery power. See “man pacemaker” or do a little searching on the web. By itself, the concept of pacemaker doesn't seem like it should make a huge different on clock synchronization, however, Apple actually moved the responsibility for adjusting the clock completely into pacemaker and removed that functionality from the MacOSX 10.9.x ntpd distribution. This has turned out to be problematic.

Mac OSX 10.9.x has a modified NTP Daemon

The Mac OSX 10.9.x ntpd will query servers as normal, derive the computer clock’s drift, update the ntp.drift file. This actually appears to work, at least for some short period of time after ntpd is restarted. However, Mac OSX ntpd does not adjust the computer’s clock at all, because pacemaker now performs that job. Also, after a while the Mavericks ntpd simply starts rejecting all NTP servers.

Date & Time Preference Pane

  • Leads you to believe a single time server is sufficient. You can enter multiple time servers in that Preference Pane by comma-separating the values, but that doesn't allow you to include any NTP options.
  • The Preference Pane clobbers any customizations made to /etc/ntp.conf. Fortunately, you can configure NTP to your own specs quite well without ever using the Date & Time Preference Pane, so this clobbering can be avoided by ignoring the Preference Pane.

What’s wrong with NTP in Mavericks?

There's quite a bit of history on NTP problems in Mac OSX. I only noticed issues beginning with the Mac OSX 10.9.1, so I'm not considering what might have been happening in the pre-Mavericks days. If you want to read up on the reported problems, here are links to some of the more recent and relevant discussions:

There are also quite a few relevant discussions on Apple's Support forums. I've read most of them, I've tried many of them, and while some of them initially resolve clock sync issues, they don't address all of the problems that I've found.

Deep Dive on inspecting the problem

If your Mac has been running for more than a few days and you compare your Mavericks clock with a better source of time, e.g. Apple's Time Server, you’ll probably notice that your clock is different. To really quantify the difference, run this command in a Terminal window:

cygnus:~ user$ sntp time.windows.com
2014 Mar 01 19:29:53.845391 +1.1 +/- 0.046295 secs

That shows a +1.1 second offset from Apple’s time server on an unmodified 10.9.2 iMac that has been running for just 3 days. Give it a few more days and that offset will continue to grow: My 10.9.1 systems were 10's of seconds out of sync. Here’s an example of the same command on a “repaired” 10.9.2 MacBook Pro.

orion:~ user$ sntp time.apple.com
2014 Mar 02 16:59:24.9871 +0.005949 +/- 0.005661 secs

In this case, the offset is less than 6 milliseconds...much better. Still not convinced? Want to see some other debug output from an unrepaired 10.9.2 system?

cygnus:~ user$ ntpq -c pe -c as
     remote           refid      st t when poll reach   delay   offset  jitter
 x-plane.waitone      3 u  826   64    3    0.282   25.263  43.154    2 u  763   64    1   69.603   41.587  34.588
 tock2.usshc.com .GPS.            1 u  698   64    1  109.476    6.248  16.566
 falcon.ca.us.sl    2 u  891   64    1   86.285   35.751  41.774
 xen1.rack911.co    3 u  889   64    1   84.967   38.992  38.525

ind assid status  conf reach auth condition  last_event cnt
  1 34353  9014   yes   yes  none    reject   reachable  1
  2 34354  9034   yes   yes  none    reject   reachable  3
  3 34355  9014   yes   yes  none    reject   reachable  1
  4 34356  9014   yes   yes  none    reject   reachable  1
  5 34357  9034   yes   yes  none    reject   reachable  3

That output uses ntpq ("ntp query") to show ntpd’s current state for each of the associated servers. All but the first server in the list are from the 0.us.pool.ntp.org NTP server pool. What’s worth noting is that ntpd has rejected every potential NTP server. The Apple-modified ntpd has given up and isn’t peering with any servers. (The observant user will notice that I did modify /etc/ntp.conf to include additional servers beyond the default.) Here’s the same command from a repaired 10.9.2 installation:

orion:~ user$ ntpq -4c pe -c as
     remote           refid      st t when poll reach   delay   offset  jitter
-watt.22pf.org  2 u  391 1024  377   22.236    8.664   2.516
+ns1.technotic.c  3 u  943 1024  377   18.286    6.770   3.498
*vcxen02.bw.phl.   2 u   63 1024  377   15.712    7.468   0.797
+ntp1.housing.be    2 u 1971 1024  376   93.100    6.204   2.429
-mail.honeycomb.    3 u  280 1024  377   42.294    5.763   7.698
-ccadmin.cycores   2 u  191 1024  377   29.587    4.147   2.553
-sisdb01.muskego  3 u  731 1024  377   47.663   10.910   6.139
-fush.slash31.co  2 u   29 1024  377   85.637   11.146   1.814
-time.apple.com   2 u  320 1024  377   22.112    2.890   3.932

ind assid status  conf reach auth condition  last_event cnt
  1 57364  933a   yes   yes  none   outlyer    sys_peer  3
  2 57365  9424   yes   yes  none candidate   reachable  2
  3 57366  961a   yes   yes  none  sys.peer    sys_peer  1
  4 57367  943a   yes   yes  none candidate    sys_peer  3
  5 57368  9324   yes   yes  none   outlyer   reachable  2
  6 57369  931a   yes   yes  none   outlyer    sys_peer  1
  7 57370  9324   yes   yes  none   outlyer   reachable  2
  8 57371  931a   yes   yes  none   outlyer    sys_peer  1
  9 57372  931a   yes   yes  none   outlyer    sys_peer  1

That is a sight for sore eyes! There are a few candidates, a number of outlyers (sic), no rejects, and ntpd has found a good peer. Accurate clock sync is achievable in Mavericks!

If you want an accurate clock in Mavericks    

This is not for the feint of heart: These problematic components are part of the OS. I tried a number of different ways to fix the existing components, but have simply not found a way to make it work. This is transplant time, so scrub up and prep for surgery!    

Outline of the work

  1. Shutdown pacemaker - it's getting in the way and we don't need it. ntpd has been optimized over the years to minimize its impact on the host computer. My computers run on AC 99% of the time, so even if pacemaker does help with battery life, it’s not a factor for me.    
  2. Replace the built-in ntpd and associated utilities - Apple's ntpd distribution is hobbled, plain and simple. You can leave the existing executables in place if you’d like, they will just not be used.    
  3. Improve on the default NTP configuration by using NTP pools and just one additional option in /etc/ntp.conf.

Shutdown pacemaker    

We're going to turn off pacemaker, put it on the shelf, and leave it there for safe keeping. We don't want it. Pacemaker is managed by launchd, which in turn is managed by the launchctl command. In the terminology of launchctl, you need to unload the pacemaker configuration.

cygnus:LaunchDaemons user$ cd /System/Library/LaunchDaemons
cygnus:LaunchDaemons user$ sudo launchctl unload com.apple.pacemaker.plist

Pacemaker will never rear its head again, unless you “load” it manually.    

Download, compile, and install Clean NTP Binaries    

OK, we're now going to dive a little deeper, but the NTP distribution is relatively easy to build.

  1. You will need Xcode. It's in the Mac App Store and you've already paid for it by buying an Apple computer.
  2. Download the latest Production version of NTP from the NTP Source Distribution website.
  3. Build and install NTP.
  4. You should wind up with these nice new executables in /usr/local/bin:
orion:~ user$ ls /usr/local/bin
ntp-keygen  ntpd     ntpdc  ntpsnmpd  sntp
ntp-wait    ntpdate  ntpq   ntptrace  tickadj

Modify /usr/libexec/ntpd-wrapper to use the Clean NTP binaries.    

ntpd-wrapper is executed by launchd. During boot up, ntpd-wrapper first executes sntp to make coarse clock adjustments, then starts ntpd which will run forever and make fine adjustments to the system clock. The original sntp and ntpd executables are still loaded on your computer and, unless you modify root’s executable search path, they will be executed by default. We need to ensure that ntpd-wrapper executes the new Clean NTP binaries.

  1. Using your favorite editor (vim is more than up to this task), look for “if sntp” and “exec ntpd” in /usr/libexec/ntpd-wrapper.    
  2. Prepend “/usr/local/bin/“ to sntp and ntpd in /usr/libexec/ntpd-wrapper.

The last 9 lines of ntpd-wrapper should look like this when you're done:

for server in $(awk '/^server/ {print $2}' /etc/ntp.conf); do

if /usr/local/bin/sntp -K /dev/null -s ${server} &> ${LOG}; then
    logger -p daemon.err -f ${LOG}


exec /usr/local/bin/ntpd -c /private/etc/ntp-restrict.conf -n -g -p /var/run/ntpd.pid -f /var/db/ntp.drift `

(Need a screen shot here)

Improve the default /etc/ntp.conf    

You have many options available to you at this point. I personally include time.apple.com in my configuration. The NTP Pool project is a terrific clearinghouse of public NTP servers, so I’ve added that to the mix. I’ve also experimented with using time.nist.gov, but this particular machine isn’t using that right now.

driftfile /var/db/ntp.drift    
pool 0.us.pool.ntp.org iburst    
pool 1.us.pool.ntp.org iburst    
server time.apple.com iburst

This configuration works very nicely.    

Finally, let’s restart using the new ntpd. But before you restart, how do you know whether you’re using old or new versions?

cygnus:bin user$ pgrep -lf ntpd    
116 /usr/sbin/ntpd -c /private/etc/ntp- restrict.conf -n -g -p /var/run/ntpd.pid -f /var/db/ntp.drift

Notice “/usr/sbin/ntpd”, that’s the original ntpd.    

Restart like this:    

cygnus:bin user$ sudo launchctl stop org.ntp.ntpd    
cygnus:bin user$ sudo launchctl start org.ntp.ntpd    
cygnus:bin user$ pgrep -lf ntpd    
11145 /usr/local/bin/ntpd -c /private/etc/ntp-restrict.conf -n -g -p /var/run/ntpd.pid -f /var/db/ntp.drift

Notice "/usr/local/bin/ntpd" and recall that's the new clean version that you just built and installed.

At this point, ntpd will begin building up statistics about each of the potential servers and will eventually decide to peer with one of them. Use the ntpq commands that I demonstrated earlier to inspect the synchronization state. Monitor using ntpq and sntp (or ntpdate) over the next few hours and days to verify that your Mac:

  1. does not reject all (and probably not any of) the time servers
  2. ntpd and your computer's clock gradually improves time sync with respect to the NTP servers.

Don’t expect sub-millisecond accuracy when synchronizing across the Internet like this, but you can expect the clock to be within 10-20 milliseconds most of the time. Three of my four Macs have an estimated offset of less than 10 milliseconds, while the fourth is running totally "open loop" with Apple's ntpd...it's marching to its own beat!

This article seems very long, but it's really scraping the surface of the Network Time Protocol. I originally went through this effort with Mac OSX 10.9.1 and verified that these fixes work (and are still necessary) with Mac OSX 10.9.2. If you're having computer clock issues on your Mavericks Mac, as was I, these steps are the only way I've found to get your computer's clock on straight and narrow. Good luck!