When the Navy Master Clock Fails

“A man with a watch knows what time it is. A man with two watches is never sure.”– Segal’s Law (Source: US Naval Observatory Website)

R/GA wrote the software that synchronizes the playback of more than a dozen mega-signs in Times Square. Network Time Protocol (NTP) is a big part of what keeps the signs operating in lockstep; NTP enables sign computers to maintain system time synchronization to the US Naval Observatory Master Clock in Washington, DC.

For more than ten years the system worked without a hiccup, but that changed on November 19 when the sign clocks began to drift mysteriously. After an extensive code review, log file analysis and several fruitless detours, the trail eventually lead to a very unexpected place: USNO.NAVY.MIL.

It turned out a resource we really thought we could count on failed. For fifty-one minutes on November 19th, the US Naval Observatory Master Clock gave out the year 2000 instead of 2012.  While it’s true it’s good practice to source NTP from additional sources (the pools at ntp.org for example),  the navy uses a pool too (not to mention super accurate cesiums and H-masers) and when confronted with a once in a decade situation, what system architect is really ready to pick 0.north-america.pool.ntp.org over tick.usno.navy.mil?