At the STAC Summits in London and New York a few weeks ago, time and time synchronization were featured prominently. The recent tightening of timestamp reporting under MiFID II regulations, and the anticipation of similar regulatory developments in other jurisdictions, is one reason for the heightened interest in time synchronization in the trading industry.
My colleague James Wylie and I had the opportunity to share some of the experiences that Corvil and its customers have had with Precision Time Protocol (PTP), also known by its IEEE standards number 1588. As its name suggests, PTP is a network protocol designed to distribute a time signal in a highly precise manner. It is often contrasted with NTP, or Network Time Protocol, a much older but still-evolving RFC-based standard which is the mechanism of choice for distributing time across the Internet. The simple contrast that is drawn can be summarised as "PTP provides much more accurate time than NTP". However, the full story is a lot more complex than this. First of all, NTP and PTP have quite different design goals, with NTP being more focused on robustness, internet scale, and security than on raw accuracy. Also, NTP is quite a flexible framework and deployments of NTP can be tuned to achieve surprisingly precise levels of clock synchronization; Conversely PTP can prove to be a tricky beast to deploy reliably, as James and I related in our presentations at the STAC summits.
However, there is more than a grain of truth to the simple contrast, and there is one key design feature of PTP that makes it capable of far greater levels of precision than NTP, namely hardware timestamping. To understand the role that hardware can play, let's review for a moment the basic principle that underlies NTP, PTP, and any other network-based time-distribution protocol. Here's a diagram showing the mechanism that NTP uses:
Fig. 1: Lifecycle of an NTP poll.
The server on top has an accurate clock, and the client below wants to read it so it can synchronize its own clock. The client sends a request at time t0 which arrives at the server at time t1; the server handles the request and at time t2 sends its response, which finally arrives back to the client at time t3. PTP rearranges the order of these events, decoupling the client request from the server response so they can happen at different times. However the basic idea is the same and, in particular, the raw calculation the client does to start figuring out the offset of its clock from that of the server is the same:
Two critical observations about this calculation are:
If the clocks were perfectly synchronized, then t1 - t0 is just the network delay in the client-server direction, and t3 - t2 is the server-client network delay.
In fact, both NTP and PTP make the assumption that the basic propagation delay in both directions across the network is the same, and use the raw offset calculation above to get an initial estimate of the actual offset of the client clock and correct it.
The really tricky part in network timing is that the network delay is noisy. Even if the client clock is perfectly synchronized, transient differences in the forward and reverse delays can trick the naive offset calculation above into believing it requires a correction.
This inevitable noise in network communication is often the determining factor in how well different implementations of NTP and PTP can perform: those that do a better job of filtering out the noise can synchronize system clocks more accurately.
However it's also here that PTP has a trick up its sleeve in the form of hardware timestamping: with NTP, the noise in the network delay is the aggregate of all the noise along each hop of the network between the client and the server. With PTP, we get to put an accurate clock on each interface in the network, both hosts and switches, so that a clean time signal can be transferred through the network without any impact from queuing delay or other forms of jitter inside network devices. (How exactly that works is a topic for another time...)
Although most discussions of PTP focus on its operation inside the network, its edge over NTP comes at least as much from its ability to sidestep jitter in the network stack inside the end hosts themselves. In NTP, timestamps are read in software and written to a UDP socket, which means that they need to go through the host UDP stack and NIC driver before even starting their journey across the network. We did some experiments to illustrate this:
Fig. 2: UDP stack latency for timestamp queries and replies
Figure 2 shows the UDP stack latency for NTP-style timestamp queries and replies, with the client-to-server query latency shown in green, and the response latency shown in red. We have presented the results as a time-series to show that the behaviour, while noisy, is fairly consistent: the stack jitter is typically several dozens of microseconds, and sometimes up to 200us.
This is the level of noise that NTP has to contend with, in contrast to PTP which, through the use of hardware timestamping, can completely bypass all this noise in its raw data.
PTP, by using hardware timestamping, can completely eliminate dozens of microseconds of UDP stack jitter that NTP has to contend with in software timestamps, which gives it a big headstart in the process of accurately synchronizing the clocks in network-attached hosts.