21 comments

  • dmazin3 hours ago
    I highly recommend anyone to look up how PTP works and how it compares to NTP. Clock sync is very interesting. When I joined an HFT company, first thing I did was understand this stuff. We care about it a lot[1].<p>If you want a specific question to answer, answer this: why does PTP need hardware timestamping to achieve high precision (where the network card itself assigns timestamps to packets, rather than having the kernel do it as part of TCP&#x2F;IP processing)? If we use software timestamps, why can we do microsecond precision at best? If you understand this, it goes a very long way to understanding the core ideas behind precise clock sync.<p>Once you have a solid understanding of PTP, look into White Rabbit. They’re able to sync two clocks with sub-ns precision. In case that isn’t obvious, that is absolutely insane.<p>[1] So do a lot of people. For example audio engineers. Once, an audio engineer absolutely talked my ear off about ptp. I had no idea that audio people understood clock sync so well but they do!
    • RossBencina1 hour ago
      &gt; So do a lot of people. For example audio engineers.<p>Indeed. PTP (various, not-necessarily compatible, versions) is at the core of modern ethernet-based audio networking: Dante (proprietary, PTP: IEEE 1588 v1), AVB (IEEE standard, PTP: 802.1AS), AES67 (AES standard, PTP: IEEE 1588 v2). And now the scope of the AVB protocol stack has been expanded to TSN for industrial and automotive time sensitive network applications.
      • dmazin35 minutes ago
        Yeah, the audio engineer then talked my ear off about networking!
    • baby_souffle1 hour ago
      Not just audio, anybody in the live events &#x2F; production space needs all equipment marching in lock step.
  • josephg5 hours ago
    &gt; When two transactions happen at nearly the same time on different nodes, the database must determine which happened first. If clocks are out of sync, the database might order them incorrectly, violating consistency guarantees.<p>This is only true if you use wall clock time as part of your database’s consistency algorithm. Generally I think this is a huge mistake. It’s almost always much easier to swap to a logical clock - which doesn’t care about wall time. And then you don’t have to worry about ntp.<p>The basic idea is this: event A happened before event B iff A (or something that happened after A) was observed by the node that generated B before B was generated. As a result, you end up with a dag of events - kind of like git. Some events aren’t ordered relative to one another. (We say, they happened concurrently). If you ever need a global order for all events, you can deterministically pick an arbitrary order for concurrent events by comparing ids or something. And this will give you a total order that will be the same on all peers.<p>If you make database events work like this, time is a little more complex. (It’s a graph traversal rather than simple numbers). But as a result the system clock doesn’t matter. No need to worry about atomic clocks, skew, drift, monotonicity, and all of that junk. It massively simplifies your system design.
    • johnisgood3 hours ago
      Related in many ways: <a href="https:&#x2F;&#x2F;www.erlang.org&#x2F;docs&#x2F;22&#x2F;apps&#x2F;erts&#x2F;time_correction" rel="nofollow">https:&#x2F;&#x2F;www.erlang.org&#x2F;docs&#x2F;22&#x2F;apps&#x2F;erts&#x2F;time_correction</a><p>Also I still remember having fun with the &quot;Determine the order of events by saving a tuple containing monotonic time and a strictly monotonically increasing integer as follows&quot; part.
  • NelsonMinar6 hours ago
    On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools.<p>This post is about more complicated synchronization for more demanding applications. And it&#x27;s very good. I&#x27;m just marveling at how in my lifetime I from &quot;no clock is ever set right&quot; to assuming most anything was within a second of true time.
    • Uehreka2 hours ago
      I was doing something at work that involved calculating round trip times from&#x2F;to Android devices, and learned that although it should be possible for NTP to sync clocks with below-second precision, in practice many of the Android devices I was working with (mostly Pixels 2-7) were off from my server and each other by up to 5 seconds, which blew my mind.
      • winrid29 minutes ago
        Yeah it can only do so much with varying latency
    • raron3 hours ago
      &gt; clock sync for civilians has never been easier<p>I don&#x27;t think civilian clock synchronization was an issue since a long time ago.<p>DCF77 and WWVB has been around for more than 50 years. You could use some cheap electronics and get well below millisecond accuracy. GPS has been fully operational for 30 years, but it needs more expensive device.<p>I suspect you could even get below 1 sec accuracy using a watch with a hacking movement and listening to radio broadcast of time beeps &#x2F; pips.
      • ssl-338 minutes ago
        Both of the WWVB clocks I&#x27;ve owned have been very fickle about how they&#x27;re placed because RF be that way sometimes, and Colorado isn&#x27;t exactly nearby to my location in Ohio.<p>The first manufactured GPS clock I owned (as in: switch it on and time is shown on a dedicated display) was in a 2007 Honda.<p>But a firmware bug ruined that clock: <a href="https:&#x2F;&#x2F;didhondafixtheclocks.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;didhondafixtheclocks.com&#x2F;</a><p>And even after it began displaying the right time again, it had the wrong date. It was offset by years and years, which was OK-ish, but also by several months.<p>Having the date offset by months caused the HVAC to behave in strange incurable ways because it expected the sun to be in positions where it was not.<p>But NTP? NTP has never been fickle for me, even in the intermittently-connected dialup days I experienced ~30 years ago: If I can get to the network occasionally, then I can connect to a few NTP servers and keep a local clock reasonably-accurate.<p>NTP has been resolutely awesome for me.
    • jasonwatkinspdx5 hours ago
      At this point the only clock in my life that doesn&#x27;t auto set is the one on my stove, and that&#x27;s because I abhor internet connected kitchen appliances.
      • timschmidt2 hours ago
        Good ol&#x27; Oven Standard Time (OST).
      • amelius4 hours ago
        Same here. I wish there was an easy way around it (that doesn&#x27;t require me to play sysadmin in my spare time).
        • jaggederest4 hours ago
          In the 80s my uncle had digital clocks that used an antenna to tune into the atomic clock time signal that (was&#x2F;is?) broadcast nationwide. I&#x27;ve long wished that it was incorporated into stoves, microwaves, essentially everything that isn&#x27;t an internet device (yet... sigh)<p>Sadly I think the actual antenna and hardware were relatively large since it&#x27;s a long wave signal, but maybe with SDR it&#x27;ll all fit on the head of a pin these days.
          • jasonwatkinspdx3 hours ago
            &gt; Sadly I think the actual antenna and hardware were relatively large since it&#x27;s a long wave signal, but maybe with SDR it&#x27;ll all fit on the head of a pin these days.<p>Unfortunately there&#x27;s no real way to cheat physics as far as shrinking a wavelength goes. With RF antennas about the best you can do is a major dimension 1&#x2F;10th the frequency of interest.
          • raron3 hours ago
            &gt; atomic clock time signal that (was&#x2F;is?) broadcast nationwide<p>Probably DCF77 or WWVB.<p>&gt; I think the actual antenna and hardware were relatively large since it&#x27;s a long wave signal<p>Casio has some normal sized wristwatches that synchronizes to DCF77, it would definitely fit into a stove, microwave, or basically anything.
            • jaggederest2 hours ago
              I believe it was a longwave broadcast so probably WWVB which would apparently imply a 60mm antenna, but it was a standard old school &quot;GE digital clock radio&quot; form factor so size wasn&#x27;t at a premium.
  • j_seigh7 hours ago
    Ok,so people use NTP to &quot;synchronize&quot; their clocks and then write applications that assume the clocks are in exact sync and can use timestamps for synchronization, even though NTP can see the clocks aren&#x27;t always in sync. Do I have that right?
    • kccqzy7 hours ago
      If you are an engineer at Google dealing with Spanner, then you <i>can</i> in fact assume clocks are well synchronized and can use timestamps for synchronization. If you get commit timestamps from Spanner you can compare them to determine exactly which commit happened first. That’s a stronger guarantee than the typical Serializable database like postgresql: <a href="https:&#x2F;&#x2F;www.postgresql.org&#x2F;docs&#x2F;current&#x2F;transaction-iso.html#XACT-SERIALIZABLE" rel="nofollow">https:&#x2F;&#x2F;www.postgresql.org&#x2F;docs&#x2F;current&#x2F;transaction-iso.html...</a><p>That’s the radical developer simplicity promised by TrueTime mentioned in the article.
      • vlovich1237 hours ago
        That’s actually not at all what TrueTime guarantees and assuming they’ve solved a physical impossibility is dangerous technically as a founding assumption for higher level tech (which thankfully Spanner does not do).<p>What TrueTime says is that clocks are synchronized within some delta just like NTP, but that delta is significantly smaller thanks to GPS time sync. That enables applications to have tighter bounds on waiting to see if a conflict may exist before committing which is why Spanner is fast. CockroachDB works similarly but given the logistical challenge of getting GPS receivers into data centers, they worked to achieve a smaller delta through better NTP-like timestamps and generally get fairly close performance.<p><a href="https:&#x2F;&#x2F;programmingappliedai.substack.com&#x2F;p&#x2F;what-is-true-time-and-how-it-is-used" rel="nofollow">https:&#x2F;&#x2F;programmingappliedai.substack.com&#x2F;p&#x2F;what-is-true-tim...</a><p>&gt; Bounded Uncertainty: TrueTime provides a time interval, [earliest, latest], rather than a single timestamp. This interval represents the possible range of the current time with bounded uncertainty. The uncertainty is caused by clock drift, synchronization delays, and other factors in distributed systems.
        • kccqzy7 hours ago
          That’s exactly what I’m saying but you simply provided more details. TrueTime guarantees clocks are well synchronized: and of course that means synchronized to a reasonable upper bound. It’s no more possible for clocks to be absolutely synchronized, than for two line segments drawn independently to have absolutely the same length.
          • vlovich1237 hours ago
            &gt; you can compare them to determine exactly which commit happened first<p>This is the part I was referring to. You cannot just compare timestamps and know which happened first. You have to actually handle the case where you don’t know if there’s a happens before relationship between the timestamps. Thats a very important distinction
            • kccqzy3 hours ago
              I quote from Spanner docs at <a href="https:&#x2F;&#x2F;docs.cloud.google.com&#x2F;spanner&#x2F;docs&#x2F;true-time-external-consistency" rel="nofollow">https:&#x2F;&#x2F;docs.cloud.google.com&#x2F;spanner&#x2F;docs&#x2F;true-time-externa...</a><p>&gt; External consistency states that Spanner executes transactions in a manner that is indistinguishable from a system in which the transactions are executed serially, and furthermore, that the serial order is consistent with the order in which transactions can be observed to commit. Because the timestamps generated for transactions correspond to the serial order, if any client sees a transaction T2 start to commit after another transaction T1 finishes, the system will assign a timestamp to T2 that is higher than T1&#x27;s timestamp.<p>Of course there is always the edge case where two commits have the same commit timestamp. Therefore from the perspective of Spanner, they happen simultaneously and there is no way to determine which happens first. But there is no need to. There is no causality relationship between them. If you insist, you can arbitrarily assign a happens-before relationship in your own code and nothing will break.
      • idorosen7 hours ago
        Alternatively, you could guarantee the same synchronization using PPS and PTP to each host&#x27;s DCD pin of their serial port or to specialized hardware such as modern PTP-enabled smart NICs&#x2F;FPGAs that can accept PPS input. GPS+PPS gets you to within 20-80ns global synchronization depending on implementation (assuming you&#x27;re all mostly in the same inertial frame), and allows you to make much stronger guarantees than TrueTime (due to higher precision distributed ordering guarantees, which translate to lower latency and higher throughput distributed writes).
        • structural4 hours ago
          Of course, you can do this in good conditions. The extremely powerful part that TrueTime brings is how the system degrades when something goes wrong.<p>If everyone is synced to +&#x2F;- 20ns, that&#x27;s great. Then when someone flies over your datacenter with an GPS jammer (purposeful or accidental), this needs to not be a bad day where suddenly database transactions happen out of order, or you have an outage.<p>The other benefit of building in this uncertainty to the underlying software design is you don&#x27;t have to have your entire infrastructure on the same hardware stack. If you have one datacenter that&#x27;s 20yrs old, has no GPS infrastructure, and operates purely on NTP - this can still run the same software, just much more slowly. You might even keep some of this around for testing - and now you have ongoing data showing what will happen to your distributed system if GPS were to go away in a chunk of the world for some sustained period of time.<p>And in a brighter future, if we&#x27;re able to synchronize everyone&#x27;s clocks to +&#x2F;- 1ns, the intervals just get smaller and we see improved performance without having to rethink the entire design.
          • throw0101c2 hours ago
            &gt; <i>Then when someone flies over your datacenter with an GPS jammer (purposeful or accidental), this needs to not be a bad day where suddenly database transactions happen out of order, or you have an outage.</i><p>Most NTP&#x2F;PTP appliances have internal clocks that are OCXO or rubidium that have holdover (even for several days).<p>If time is that important to you then you&#x27;ll have them, plus perhaps some fibre connections to other sites that are hopefully out of range of the jamming.
            • RossBencina1 hour ago
              &gt; fibre connections to other sites that are hopefully out of range of the jamming.<p>I guess it&#x27;s not inconceivable that eventually there&#x27;s a global clock network using a White-Rabbit-like protocol over dedicated fibre. But if you have to worry about GPS jamming you probably have to worry about undersea cable cutting too.
          • idorosen4 hours ago
            In summary, with different business requirements you would build a different technical solution.
        • vlovich1237 hours ago
          &gt; and allows you to make much stronger guarantees than TrueTime (due to higher precision distributed ordering guarantees, which translate to lower latency and higher throughput distributed writes).<p>TrueTime is the software algorithm for managing the timestamps. It’s agnostic to the accuracy of the underlying time source. If it was inaccurate then you get looser bounds and as you note higher latency. Google already does everything you suggest for TrueTime while also having atomic clocks in places.
          • idorosen4 hours ago
            Yup! I was referring to the original TrueTime&#x2F;Spanner papers, not whatever&#x27;s currently deployed. The original paper makes reference to distributed ordering guarantees at the milliseconds&#x27; scale precision, which implies many more transactions in flight in the uncertain state and coarser distributed ordering guarantees than the much tighter upper bound you can set with nanoseconds&#x27; precision and microseconds&#x27; comms latency...
        • AlotOfReading7 hours ago
          Truetime is based on GPS and local atomic clocks. Google&#x27;s latest timemasters are even better, around 10ns average.
      • awesome_dude7 hours ago
        Isn&#x27;t that because Google has its own atomic clocks, rather than NTP which is (generally) using publicly available atomic clocks?
        • vlovich1236 hours ago
          More that they use GPS to synchronize the clocks. Having your own atomic clock doesn’t really improve your accuracy except for within the single data center you have it deployed (although I’m sure there’s techniques for synchronizing with low bounds against nearby atomic clocks + GPS to get really tight bound so they don’t need one in every data center)
    • TrainedMonkey7 hours ago
      Depending on the application you would generally use PTP to get sub-microsecond accuracy. The real trick is that architecture should tolerate various clocks starting or jumping out of sync and self correct.
    • goodpoint4 hours ago
      *misuse timestamps for synchronization
  • georgelyon8 hours ago
    Unfortunate that the author doesn’t bring up FoundationDB version stamps, which to me feel like the right solution to the problem. Essentially, you can write a value you can’t read until after the transaction is committed and the synchronization infrastructure guarantees that value ends up being monotonically increasing per transaction. They use similar “write only” operations for atomic operations like increment.
    • lll-o-lll7 hours ago
      Yes. A consistent total ordering is what you need (want) in distributed computing. Ultimately, causality is what is important, but consistent ordering of concurrent operations makes things <i>much</i> easier to work with.
      • josephg5 hours ago
        Consistent ordering of concurrent operations is easy though. Just detect this case (via logical clocks) then order using node ids or transaction ids if the logical clocks show the transactions as being concurrent. Am I missing something? This feels like a very solved problem. (I’ve worked on CRDTs where we have the same problem. There exist incredibly fast algorithms for this.)
        • lll-o-lll2 hours ago
          &gt; Am I missing something?<p>I don’t think so, I think it is solved in the general sense. However what Spanner does <i>is</i> unique, and it <i>does</i> use synchronised clocks in order to do it.<p>However, Spanner does not solve the inter-continental acid database with high write throughput. So I don’t see it as ground breaking. CRDT’s are interesting, I’ve followed your work for a long time, but too constrained to solve this general problem I think.
        • georgelyon2 hours ago
          Yes, though the API of having a write-only value that is a monotonically increasing counter is much simpler than having to think about causality or logical clocks.
  • pdeva11 hour ago
    AWS has the Google TrueTime equivalent precision clock available for public use[1] which makes this problem much easier to solve now. Auora DSQL uses it. Even third party db&#x27;s like YugabyteDb make use of it.<p>[1] <a href="https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;AWSEC2&#x2F;latest&#x2F;UserGuide&#x2F;set-time.html" rel="nofollow">https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;AWSEC2&#x2F;latest&#x2F;UserGuide&#x2F;set-time...</a>
  • gdcohen1 hour ago
    Take a look at Exploiting a Natural Network Effect for Scalable, Fine-grained Clock Synchronization - <a href="https:&#x2F;&#x2F;www.usenix.org&#x2F;conference&#x2F;nsdi18&#x2F;presentation&#x2F;geng" rel="nofollow">https:&#x2F;&#x2F;www.usenix.org&#x2F;conference&#x2F;nsdi18&#x2F;presentation&#x2F;geng</a> (commercial version by the same authors - clockwork.io).
  • didgetmaster2 hours ago
    Reminds me of the old saying: &#x27;If you have just one watch&#x2F;clock, then you always know what time it is; but if you have two of them, then you are never sure!&#x27;
  • kobieps8 hours ago
    Even just a single accurate clock is a nightmare... <a href="https:&#x2F;&#x2F;www.npr.org&#x2F;2025&#x2F;12&#x2F;21&#x2F;nx-s1-5651317&#x2F;colorado-us-official-time-microseconds-nist-clocks" rel="nofollow">https:&#x2F;&#x2F;www.npr.org&#x2F;2025&#x2F;12&#x2F;21&#x2F;nx-s1-5651317&#x2F;colorado-us-off...</a>
  • Asmod4n6 hours ago
    Does wall clock time matter for anything but logging? For everything else one could just create any form „time“ to keep stuff in sync, no?
    • tbrownaw6 hours ago
      Isn&#x27;t it also useful for checking validity periods for stuff like TLS certs or JWTs or Kerberos tickets?
      • Asmod4n6 hours ago
        We could use some made up „time“ for that since it’s not made for human consumption just sync between different systems.
        • tbrownaw5 hours ago
          I suppose this is changing with TLS certs moving towards ephemerality, but they used to have an entry on someone&#x27;s calendar for renewal.
  • mgaunard8 hours ago
    Another protocol that&#x27;s not mentioned is PPS and its variants, such as WhiteRabbit.<p>A regular pulse is emitted from a specialized high-precision device, possibly over a specialized high-precision network.<p>Enables picosecond accuracy (or at least sub-nano).
    • nuccy4 hours ago
      As a user of WhiteRabbit, I can confirm a sub-10ps sync (two clocks phase lock) over 50km fiber connection for variable temperature of fiber (biggest problem of clock sync over fibers is temperature induced length change of the fiber itself, which needs to be measured and compensated).
      • RossBencina1 hour ago
        Out of interest, how do you measure a sub-10ps phase lock between devices 50km apart?
        • willis93647 minutes ago
          The standards-compliant endpoints do all of the work. They count clock cycles for ping pong messages and share with each other the length of time so time-of-flight is tracked and compensated for.
  • maximinus_thrax8 hours ago
    I wouldn&#x27;t say it&#x27;s a &#x27;nightmare&#x27;. It&#x27;s just more complicated than what regular folk think computers work when it comes to time sync. There&#x27;s nothing nightmareish or scary about this, it&#x27;s just using the best solution for your scenario, understanding limitations and adjusting expectations&#x2F;requirements accordingly, perhaps relaxing consistency requirements.<p>I worked on the NTP infra for a very large organization some time ago and the starriest thing I found was just how bad some of the clocks were on &#x27;commodity hardware&#x27; but this just added a new parameter for triaging hardware for manufacturer replacement.<p>This is an ok article but it&#x27;s just so very superficial. It goes too wide for such a deep subject matter.
    • senfiaj4 hours ago
      Maybe. But I remember one game developer told that they face even a more challenging problem, which is the synchronization between players in multiplayer real-time games. Just imagine different users having significantly different network latencies in a multiplayer shooter where a couple milliseconds can be decisive. Someone makes a headshot when the game state is already outdated. If you think about this you can appreciate how it&#x27;s complicated just to make the gameplay at least not awful...
    • hinkley7 hours ago
      I took to distributed systems like a duck to water. It was only much later that I figured out that while there are things I can figure out in one minute that took other people five, there were a lot of others that you will have to walk them through step by step or they would <i>never</i> get there. That really explained some interactions I’d had when I was younger.<p>In particular I don’t think the intuitions necessary to do distributed computing well would come to someone who snoozed through physics, who never took intro to computer engineering.
      • libraryofbabel7 hours ago
        &gt; I don’t think the intuitions necessary to do distributed computing well would come to someone who snoozed through physics<p>Yeah. I was a physics major and it really helped to have had my naive assumptions about time and clocks completely demolished early on by taking classes in special and general relatively. When I eventually found my way into tech a lot of distributed systems concepts that are difficult to other people (clock sync, indeterminate ordering of events, consensus) came quite naturally because of all that early training.<p>I think it&#x27;s no accident that distributed systems theory guru Leslie Lamport had written an unpublished book on General Relativity before he wrote the famous <i>Time, Clocks and the Ordering of Events in a Distributed System</i> paper and the Paxos paper. In the former in particular the analogy to special relatively is quite plain to see.
    • blibble8 hours ago
      PTP isn&#x27;t even that much more difficult, as long as you planned for it form the start<p>you buy the hardware, plug it all in, and it works
      • geerlingguy8 hours ago
        Sometimes hardware that has PTP support in the specs doesn&#x27;t perform very well though, so if you do things at scale, being able to validate things like switches and network card drivers is useful too!<p>It&#x27;s to the point timing server vendors I&#x27;ve spoken to have their own test labs where they have to validate network gear and then publish lists of recommended and tested configurations.<p>Even some older cards where you&#x27;d think the PTP issues would be solved still have weird driver quirks in Linux!
  • emptybits9 hours ago
    Normally I would nod at the title. Having lived it.<p>But I just watched&#x2F;listened to a Richard Feynmann talk on the nature of time and clocks and the futility of &quot;synchronizing&quot; clocks. So I&#x27;m chuckling a bit. In the general sense, I mean. Yes yes, for practical purposes in the same reference frame on earth, it&#x27;s difficult but there&#x27;s hope. Now, in general ... synchronizing two clocks is ... meaningless?<p><a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=zUHtlXA1f-w" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=zUHtlXA1f-w</a>
    • hinkley8 hours ago
      Einstein was worried about whether people in two different relativistic frames would see cause and effect reversed.
      • emptybits8 hours ago
        Wild. My layperson mind goes to a simple example, which may or may not be possible, but please tell me if this is the gist:<p>Alice and Bob, in different reference frames, both witness events C and D occurring. Alice says C happened before D. Bob says D happened before C. They&#x27;re both correct. (And good luck synchronizing your watches, Alice and Bob!)
        • hinkley8 hours ago
          Yes that definitely happens. People orbiting Polaris would be seeing two supernovas explode at different times than us due to the speed of light. Polaris is 400 light years away so the gap could be large.<p>But when you are moving you may see very closely spaced events in different order, because you’re moving toward Carol but at an angle to Doug. Versus someone else moving toward Doug at an angle to Carol.
        • mrkeen8 hours ago
          That will be the case when Alice stands close to where C happens, and Bob stands close to where D happens.<p>It&#x27;s a little trickier to imagine introducing cause-and-effect though. (Alice sees that C caused D to happen, Bob sees that D caused C to happen).<p>I think a &quot;light cone&quot; is the thought-experiment to look up here.
          • ianburrell7 hours ago
            There is distinction between seeing when events happened, and when they really happened. The latter can be reconstructed by an observer.<p>In special relativity, time is relative and when things actually happened can be different in different frames. Casually linked events are always really in the same order. But disconnected events can be seen in different orders depending on speed of observer.
            • emptybits6 hours ago
              &gt; But disconnected events can be seen in different orders depending on speed of observer.<p>What are &quot;disconnected events&quot;? In a subtle but still real sense, are not all events causally linked? e.g. gravitationally, magnetically, subatomically or quantumly?<p>I can understand that our simple minds and computational abilities lead us to consider events &quot;far away&quot; from each other as &quot;disconnected&quot; for practical reasons. But are they really not causally connected in a subtle way?<p>There are pieces of space time that are clearly, obviously causally connected to each other. And there are far away regions of the universe that are, practically speaking, causally disconnected from things &quot;around here&quot;. But wouldn&#x27;t these causally disjoint regions overlap with each other, stringing together a chain of causality from anywhere to anywhere?<p>Or is there a complete vacuum of insulation between some truly disconnected events that don&#x27;t overlap with any other observational light cone or frame of reference at all?
              • hinkley6 hours ago
                We now know that gravity moves at the speed of light. Imagie that you aretwo supernovas that for some unknown reason, explode at essentially the same time. Just before you die from radiation exposure, you will see the light pulse from each supernova before each supernova can &#x27;see&#x27; the gravitational disruption caused by the other. Maybe a gravity wave can push a chain reaction on the verge of happening into either a) happening or b) being delayed for a brief time, but the second explosion happened before the pulse from the first could have arrived. So you&#x27;re pretty sure they aren&#x27;t causally linked.<p>However if they were both triggered by a binary black hole merger, then they&#x27;re dependent events but not on each other.<p>But I think the general discussion is more of a &#x27;Han shot first&#x27; sort. One intelligent system reacting to an action of another intelligent system, and not being able to discern as a person from a different reference frame as to who started it and who reacted. So I suppose when we have relativistic duels we will have to preserve the role of the &#x27;second&#x27; as a witness to the events. Or we will have to just shrug and find something else to worry about.
              • ianburrell6 hours ago
                Causality moves at the speed of light. Events that are farther apart are called spacelike and aren&#x27;t causally connected.<p>I think you might be confusing events that have some history between them, and those are influence each other. Like say right now, Martian rover sends message to Earth and Earth sends message to them, those aren&#x27;t causally connected cause don&#x27;t know about the other message until light speed delay has passed.
              • thaumasiotes4 hours ago
                &gt; But wouldn&#x27;t these causally disjoint regions overlap with each other<p>Yes.<p>&gt; stringing together a chain of causality from anywhere to anywhere?<p>No? Causality reaching one edge of a sphere doesn&#x27;t mean it instantaneously teleports to every point in that same sphere. This isn&#x27;t a transitive relationship.<p>&gt; What are &quot;disconnected events&quot;?<p>The sentence you&#x27;re responding to seems like a decent definition. Disconnected events are events which might be observed in either order depending on the position of an observer.
          • hinkley8 hours ago
            If Bob and Alice are moving at half the speed of light in opposite directions.
    • m4638 hours ago
      it might be meaningless, but in practical terms just don&#x27;t check util.c from the gravity well into the git repo in orbit.
  • koudelka9 hours ago
    the Huygens algorithm is also worth a look<p><a href="https:&#x2F;&#x2F;www.usenix.org&#x2F;system&#x2F;files&#x2F;conference&#x2F;nsdi18&#x2F;nsdi18-geng.pdf" rel="nofollow">https:&#x2F;&#x2F;www.usenix.org&#x2F;system&#x2F;files&#x2F;conference&#x2F;nsdi18&#x2F;nsdi18...</a>
  • hinkley8 hours ago
    Vector clocks are one of the other things Barbara Liskov is known for.
  • a_t487 hours ago
    Clock sync is such a nightmare in robotics. Most OSes happily will skew&#x2F;jump to get the time correct. Time jumps (especially backwards) will crash most robotics stacks. You might decide to ensure that you have synced time before starting the stack. Great, now your timestamps are mostly accurate, except what happens when you&#x27;ve used GPS as your time source, and you start indoors? Robot hangs forever.<p>Hot take: I&#x27;ve seen this and enough other badly configured time sync settings that I want to ban system time from robotics systems - time from startup only! If you want to know what the real world time was for a piece of data after, write what your epoch is once you have a time sync, and add epoch+start time.
    • michaelt6 hours ago
      If your requirements are “must have accurate time, must start with an inaccurate time, must not step time during operation, no atomic clocks, must not require a network connection, or a WWVB signal, must work without a GPS signal” then yes, you need to relax your requirements.<p>But it doesn’t have to be the first requirement you relax.
      • a_t484 hours ago
        If it has a GPS already, it’s really easy to fall into the trap of just using it, but point taken. Then main requirement is accurate moment to moment time. Using GPS as the master clock mostly makes sense there.
    • RossBencina1 hour ago
      C++11 distinguishes system_clock from steady_clock. As you say, using system_clock is a bug.
    • awesome_dude6 hours ago
      THIS is what will save us from the robot uprising!
  • shomp5 hours ago
    Absolute synchronization impossible?? Challenge accepted.
    • nuccy4 hours ago
      Nature (laws of physics) is agains you on this: it is in fact impossible for everyone. What is in sync for some observers can be out of sync for others (depends on where they are, i.e. gravity, and how they relatively move). See general and special relativity principle of simultaneity [1].<p>1. <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Relativity_of_simultaneity" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Relativity_of_simultaneity</a>
      • Enginerrrd3 hours ago
        I think you just nerd-sniped me but I’m not convinced it’s impossible to assign a consistent ordering to events with relativistic separations.<p>For starters, the spacetime interval between two events IS a Lorentz invariant quantity. That could probably be used to establish a universal order for timelike separations between events. I suspect that you could use a reference clock, like a pulsar or something to act as an event against which to measure the spacetime interval to other events, and use that for ordering. Any events separated by a light-like interval are essentially simultaneous to all observers under that measure.<p>The problem comes for events with a space like or light like separation. In that case, the spacetime interval is still conserved, but I’m not sure how you assign order to them. Perhaps the same system works without modification, but I’m not sure.
  • forrestthewoods7 hours ago
    Timesync isn’t a nightmare at all. But it is a deep rabbit hole.<p>The best approach, imho, is to abandon the concept of a global time. All timestamps are wrt a specific clock. That clock will skew at a rate that varies with time. You can, hopefully, rely on any particular clock being monotonous!<p>My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any clock to any clock. This is a lossy conversion that has jitter and can change with time. The fewer stops the better.<p>I kinda don’t like PTP. Too complicated and requires specialized hardware.<p>This article only touches on one class of timesync. An entirely separate class is timesync within a device. Your phone is a highly distributed compute system with many chips each of which has their own independent clock source. It’s a pain in the ass.<p>You also have local timesync across devices such as wearables or robotics. Connecting to a PTP system with GPS and atomic clocks is not ideal (or necessary).<p>TicSync is cool and useful. <a href="https:&#x2F;&#x2F;sci-hub.se&#x2F;10.1109&#x2F;icra.2011.5980112" rel="nofollow">https:&#x2F;&#x2F;sci-hub.se&#x2F;10.1109&#x2F;icra.2011.5980112</a>
    • RossBencina1 hour ago
      &gt; I kinda don’t like PTP. Too complicated and requires specialized hardware.<p>In my view the specialised hardware is just a way to get more accurate transmission and arrival timestamps. That&#x27;s useful whether or not you use PTP.<p>&gt; My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any clock to any clock. This is a lossy conversion that has jitter and can change with time.<p>This sounds like the &quot;peer to peer&quot; equivalent to PTP. It would require every node to maintain state about it&#x27;s estimate (skew, slew, variance) of every other clock. I like the concept, but obviously it adds complexity to end-stations beyond what PTP requires (i.e. increases the hardware cost of embedded implementations). Such a system would also need to model the network topology, or control routing (as PTP does), because packets traversing different routes to the same host will experience different delay and jitter statistics.<p>&gt; TicSync is cool<p>I hadn&#x27;t seen this before, but I have implemented similar convex-hull based methods for clock recovery. I agree this is obviously a good approach. Thanks for sharing.
      • forrestthewoods47 minutes ago
        &gt; This sounds like the &quot;peer to peer&quot; equivalent to PTP. It would require every node to maintain state about it&#x27;s estimate (skew, slew, variance) of every other clock.<p>Well, it requires having the conversion function for each edge in the traversed path. And such function needs to exist only at the location(s) performing the conversion.<p>&gt; obviously it adds complexity to end-stations beyond what PTP requires<p>If you have PTP and it works then stick with it. If you’re trying to timesync a network of wearable devices then you don’t have PTP stamping hardware.<p>&gt; because packets traversing different routes<p>Fair callout. It’s probably a more useful model for less internty use cases. Of which there are many!<p>For example when trying to timesync a collection of different sensors on different devices&#x2F;microcontrollers.<p>Roboticists like CanBus and Ethercat. But even that is kinda overkill imho. TicSync can get you tens of microseconds of precision in user space.
    • DannyBee1 hour ago
      &quot;I kinda don’t like PTP. Too complicated and requires specialized hardware.&quot;<p>?????<p>I run PTP on everything from RPI&#x27;s to you name it, over fiber, ethernet, etc.<p>The main thing hardware gives is filtration of PTP packets or hardware timestamping.<p>Neither is actually required, though some software has decided to require it.<p>Additionally, something like 99% of sold gigabit or better chipsets since 2012 support it (I210 et al)
  • yapyap8 hours ago
    Love learning new things. This also explains why my casio clock sync starts skewing over time
  • jeffbee9 hours ago
    PTP requires support not only on your network, but also on your peripheral bus and inside your CPU. It can&#x27;t achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states.
    • pclmulqdq9 hours ago
      You can if you just run PTP (almost) entirely on your NIC. The best PTP implementations take their packet timestamps at the MAC on the NIC and keep time based on that. Nothing about CPU processing is time-critical in that case.
      • jeffbee5 hours ago
        Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. The control loop for host PTP benefits from regularity. Anyway NICs that support PTP hardware timestamping may also use PCI LTR (latency tolerance reporting) to instruct the host operating system to disable high-exit-latency sleep features, and popular operating systems respect that.
        • DannyBee1 hour ago
          &quot;Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. &quot;<p>I&#x27;m sorry, this is just moving the goalposts.<p>You said &quot;It can&#x27;t achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states.&quot;<p>This is flat wrong, as pointed out.<p>Now you are pedantically arguing that <i>some</i> NIC&#x27;s that do PTP hardware timestamping <i>might</i> also use a feature that some operating systems <i>might</i> respect.<p>That&#x27;s a very far cry from &quot;It can&#x27;t achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states&quot;.<p>In most cases, people would just say &quot;hey i was wrong about that but there are cases that i think matter where it falls down&quot;.
    • rcxdude9 hours ago
      How so? If the NIC is processing the timestamps as it arrives&#x2F;leaves on the wire, the latency and jitter in the rest of the system shouldn&#x27;t matter.
    • DannyBee1 hour ago
      PTP does not require support on your network beyond standard ethernet packet forwarding when used in ethernet mode.<p>In multicast IP mode, with multiple switches, it requires what anything running multicast between switches&#x2F;etc would require (IE some form of IGMP snopping or multicast routing or .....)<p>In unicast IP mode, it requires nothing from your network.<p>Therefore, i have no idea what it means to &quot;require support on the network&quot;.<p>I have used both ethernet and multicast PTP across a complete mishmash of brands and types and medias of switches, computers, etc, with no issues.<p>The only thing that &quot;support&quot; might improve is more accurate path delay data through transparent clocks. If both master and slave do accurate hardware timestamping already, and the path between them is constant, it is easily possible to get +-50 nanoseconds without any transparent clock support.<p>Here is the stats from a random embedded device running PTP i just accessed a second ago:<p><pre><code> Reference ID : 50545030 (PTP0) Stratum : 1 Ref time (UTC) : Sun Dec 28 02:47:25 2025 System time : 0.000000029 seconds slow of NTP time Last offset : -0.000000042 seconds RMS offset : 0.000000034 seconds Frequency : 8.110 ppm slow Residual freq : -0.000 ppm Skew : 0.003 ppm </code></pre> So this embedded ARM device, which is not special in any way, is maintaining time +-35ns of the grandmaster, and currently 30ns of GPS time.<p>The card does not have an embedded hardware PTP clock, but it does do hardware timestamp and filtering.<p>This grandmaster is an RPI with an intel chipset on it and the PPS input pin being used to discipline the chipset&#x27;s clock. It stays within +-2ns (usually +-1ns) of GPS time.<p>Obviously, holdover sucks, but not the point :)<p>This qualifies as better-than-NTP for sure, and this setup has no network support. No transparent clocks, etc. These machines have multiple media transitions involved (fiber-&gt;ethernet), etc.<p>The main thing transparent clock support provides in practice is dealing with highly variable delay. Either from mode of transport, number of packet processors in between your nodes, etc. Something that causes the delay to be hard to account for.<p>The ethernet packet processing in ethernet mode is being handled in hardware by the switches and basically all network cards. IP variants would probably be hardware assisted but not fully offloaded on all cards, and just ignored on switches (assuming they are not really routers in disguise).<p>The hardware timestamping is being done in the card (and the vast majority of ethernet cards have supported PTP harware timestamping for &gt;1 decade at this point), and works perfectly fine with deep CPU sleep states.<p>Some don&#x27;t do hardware filtering so they essentially are processing more packets that necessary but .....
  • hinkley8 hours ago
    &gt; Google faced the clock synchronization problem at an unprecedented scale with Spanner, its globally distributed database. They needed strong consistency guarantees across data centers spanning continents, which requires knowing the order of transactions.<p>&gt; Here’s a video of me explaining this.<p>Do you need a video? Do we need a 42 minute video to explain this?<p>I generally agree with Feynman on this stuff. We let explanations be far more complex than they need to be for most things, and it makes the hunt for accidental complexity harder because everything looks almost as complex as the problems that need more study to divine what is actually going on there.<p>For Spanner to be useful they needed a high transaction rate and in a distributed system that requires very tight grace periods for First Writer Wins. Tighter than you can achieve with NTP or system clocks. That’s it. That’s why they invented a new clock.<p>Google puts it this way:<p>Under external consistency, the system behaves as if all transactions run sequentially, even though Spanner actually runs them across multiple servers (and possibly in multiple datacenters) for higher performance and availability.<p>But that’s a bit thick for people who don’t spend weeks or years thinking about distributed systems.