I toggled the checkbox off then on to give NTP a little kick in the pants. After a few seconds, the clock shifted and was in sync with my other computers. Easy-peasy, right? Well, not quite.
After a few days, I noticed that the clock on this particular machine was once-again many seconds out of sync. At this point, I mentally replayed the Seinfeld episode in which Jerry reserves a car, but there's not a car to be found…
”Mac OSX knows how to set the clock sync, Mac OSX just don’t know how to maintain clock sync and that’s really the most important part, the maintaining.”
“sntp time.apple.com” uncovered that my computers were inconsistent with one another and they were not synchronized with time.apple.com. At this point, I could see how this was going to play out: Easy Peasy was not an option anymore, it was time for a deep dive into the innards of computer clock synchronization.
Now, you might look at the length of this article and think "It can't be that broken!" Trust me, I attempted to find "low impact" solutions many times based on all the existing online discussions I could find. None of them truly solved the clock sync problem for me. If you find a viable solution without requiring this level of effort, by all means, please let me know.
Network Time Protocol
Let the computer figure out what time it
The Network Time Protocol (NTP), RFC-5905, describes a technique allowing computers on the Internet to synchronize their clocks to one another. NTP defines a hierarchical NTP server structure (kind of like pyramid) composed of different strata. The higher stratum (lower numeric value) nodes are topologically closer to a high quality source of time (like a GPS receiver or maybe even an atomic clock). An individual higher stratum NTP node at stratum N will service NTP requests from a number of nodes at stratum N+1, and those stratum N+1 nodes will in turn each service requests from many stratum N+1 nodes, thus distributing the high quality time source to many lower stratum clients (like the computer in front of you), without having 30 million MacBook Pros pounding on the few high quality time servers.
NTP works in a client/server fashion, where the client and server quickly fling packets back and forth at each other, meticulously labeling each packet with the local clock time of when it was sent and received. (In reality, NTP can operate in a number of different modes, client/server being just one of them. However, it's the most used mode for home users.) By analyzing these time stamps using some well-specified math, the client can determine statistics about the network path between the client and server and the relative clock offset between the client and server.
Ignoring the complexity of how NTP arrives at this conclusion, let’s suppose the client determines that the network latency is 10 seconds (wow! I hope not!). If the server sent a packet with a server timestamp of 14:00:00.00UTC and the client received it at 14:00:12.00UTC according to the client's clock, then the client will say, “Hey, my clock is two seconds ahead of that NTP. I need to fix that!" That's my two-sentence hand-wavy explanation of NTP. I'll explain a little more of the details as I tell my story.
How many servers should you attempt to peer with? There's a parable: “A man with one watch always knows what time it is, a man with two watches is never quite sure?” I’ve seen some terribly misconfigured time servers which pop up in public NTP pools periodically: Don’t set yourself up with a single point of failure. A well-configured NTP client will attempt the synchronization exchange with a number of servers. Four or five is probably a pretty good number. The client will interrogate each time server, evaluate all of the results, and choose one of the servers as the one to trust. Again, the exhaustive detail is in the specification of NTP.
Now that the NTP client has peered with a good server, what should it do? Well, one tenet of time is that time monotonically increasing and doesn’t go backwards (at least not until we invent the flux capacitor, anyway). Going back to the example synchronization from above, NTP can’t just throw the computer’s clock in reverse back to 14:00:10.00UTC! That would mean that your computer would relive those two seconds in some alternate universe...not a good idea. Similarly, large forward jumps in time would also be disruptive and are also to be avoided. NTP needs to make small but frequent adjustments to the clock to avoid disruptive clock jumps.
Over long periods of time (minutes to hours), an NTP client continues to poll the servers and determines how many “clock ticks” the computer’s clock is drifting (in terms of parts-per-million) and writes the drift value into a file called ntp.drift. That value is different for every computer, and will even vary over time for any individual computer. Once NTP has a good drift estimate, NTP wakes up every second and makes tiny adjustments to the internal clock frequency of the computer to gradually bring it into sync and keep it there. Whew! And that’s just the glossy brochure version of NTP.
The Network Time Protocol functionality has been implemented primarily in the imaginatively named ntpd daemon and a few associated utility programs. (For information about ntpd and all things NTP, start at the NTP Project home or you can cut to the chase and go to the Network Time Synchronization Research Project.) While ntpd can utilize a number of configuration files, the interesting one is /etc/ntp.conf. Typically, that file lists the time servers that the client will interrogate. The typical M.O. for managing the life-cycle of ntpd is straightforward and relatively standard on a unix-ish system:
- Configure /etc/ntp.conf with some number of servers
- During boot time, the OS will do a few things via a one or more startup scripts:
- Execute sntp to make very gross time corrections early in the boot process
- Start ntpd with the proper command line options
- Ensure that ntpd will restart if it fails (it’s a daemon, after all)
The Mac OS X NTP Environment
Mac OSX has modified the NTP Daemon
The ntpd lifecycle is controlled by the /usr/libexec/ntpd-wrapper script. Nothing odd here since most unix operating systems have a script to manage the lifecycle of most daemon processes. More about how this works later. Unfortunately, the Mavericks ntpd-wrapper script has some problems.
- Mac OSX 10.9.1 version supported only the simplest of server configurations
- No support for “pool” entries, only “server” entries
- No support for server association options, such as the highly recommended “iburst” option. Using such features would actually cause errors at startup.
- Mac OSX 10.9.2 fixed those limitations in /usr/libexec/ntpd-wrapper.
Mavericks introduced pacemaker into the NTP ecosystem. pacemaker is a Mac OSX daemon that attempts to reduce the impact of clock synchronization on battery life. Pacemaker allows fine-grained control over the frequency with which the clock will be adjusted. Apple introduced this layer of control under the assumption that frequent updates to the computer clock will circumvent the OS’s attempts to keep the CPU idle as much as possible to save battery power. However, the ntpd community believes ntpd has little impact on power consumption and pacemaker addresses a non-issue. Pacemaker uses different adjustment intervals depending on whether running on AC or battery power. See “man pacemaker” or do a little searching on the web. By itself, the concept of pacemaker doesn't seem like it should make a huge different on clock synchronization, however, Apple actually moved the responsibility for adjusting the clock completely into pacemaker and removed that functionality from the MacOSX 10.9.x ntpd distribution. This has turned out to be problematic.
Mac OSX 10.9.x has a modified NTP Daemon
The Mac OSX 10.9.x ntpd will query servers as normal, derive the computer clock’s drift, update the ntp.drift file. This actually appears to work, at least for some short period of time after ntpd is restarted. However, Mac OSX ntpd does not adjust the computer’s clock at all, because pacemaker now performs that job. Also, after a while the Mavericks ntpd simply starts rejecting all NTP servers.
Date & Time Preference Pane
- Leads you to believe a single time server is sufficient. You can enter multiple time servers in that Preference Pane by comma-separating the values, but that doesn't allow you to include any NTP options.
- The Preference Pane clobbers any customizations made to /etc/ntp.conf. Fortunately, you can configure NTP to your own specs quite well without ever using the Date & Time Preference Pane, so this clobbering can be avoided by ignoring the Preference Pane.
What’s wrong with NTP in Mavericks?
There's quite a bit of history on NTP problems in Mac OSX. I only noticed issues beginning with the Mac OSX 10.9.1, so I'm not considering what might have been happening in the pre-Mavericks days. If you want to read up on the reported problems, here are links to some of the more recent and relevant discussions:
There are also quite a few relevant discussions on Apple's Support forums. I've read most of them, I've tried many of them, and while some of them initially resolve clock sync issues, they don't address all of the problems that I've found.
Deep Dive on inspecting the problem
If your Mac has been running for more than a few days and you compare your Mavericks clock with a better source of time, e.g. Apple's Time Server, you’ll probably notice that your clock is different. To really quantify the difference, run this command in a Terminal window: