9 comments

  • Avamander5 hours ago
    For the price and complexity of building a good Linux-based time server I&#x27;d heavily recommend looking at FPGA-based solutions instead. x86 platforms also have PCIe PTM and TGPIO options available[0], which are tad better than the common ARM equivalents and GPIO interrupts.<p>In general Pi-s are relatively bad for timekeeping. Not just the NIC (which can be replaced by hats with their own), there are also oscillator replacement mods[2] to work around this to some extent. Though all this still leaves many other sources of timekeeping error.<p>So if the goal is low-cost precise timekeeping, one can assemble something FPGA-based[2] for quite a bit less money, that can reach very low noise figures. You can spend the remaining money on the GNSS antenna and receiver instead.<p>I&#x27;m also sure a FPGA-based approach could be taken even further with some external timekeeping chips available and the FPGA could just do the NTP&#x2F;PTP protocol stuff.<p>[0]: <a href="https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=zxK0_j2Ah2I" rel="nofollow">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=zxK0_j2Ah2I</a> [1]: <a href="https:&#x2F;&#x2F;raspberrypi.stackexchange.com&#x2F;questions&#x2F;74482&#x2F;switch-out-the-x1-oscillator-on-a-rpi-2-3" rel="nofollow">https:&#x2F;&#x2F;raspberrypi.stackexchange.com&#x2F;questions&#x2F;74482&#x2F;switch...</a> [2]: <a href="https:&#x2F;&#x2F;fy.chalmers.se&#x2F;~f96hajo&#x2F;fakernet&#x2F;board_Arty_A7-35-GPS-NTP.html" rel="nofollow">https:&#x2F;&#x2F;fy.chalmers.se&#x2F;~f96hajo&#x2F;fakernet&#x2F;board_Arty_A7-35-GP...</a>
    • jeffbee5 hours ago
      Back when I thought this was true, I put together an x86 platform with PTM and a fancy NIC and it would hard-lock the bus to such an extent that I had to physically unplug it to recover. There are a lot of pieces required for a complete solution and I&#x27;d caution against a categorical statement like &quot;x86 has PTM&quot;.
      • Avamander4 hours ago
        As I said, tad better, not perfect. I&#x27;d say finding reasonably-priced x86 hardware with that functionality is an even bigger hurdle.
        • geerlingguy3 hours ago
          It all depends on the gains you&#x27;re looking for. Almost any solution with PPS input can get you under 1µs, even on Linux through GPIO.<p>If you want to chase lower, you need a hardware-disciplined clock. Either through FPGA, or in the case of my Time Pi, I have a PTP NIC with a PHC disciplined from the ublox F9T—which should be within 5-20ns of GPS time. This can be measured on higher-end boards like the TimeHAT by comparing PPS out from the GNSS module, PPS from the NIC, and PPS from a separate GPSDO, along with a timestamp comparison tool like the tinyGTC or TAPR TICC: <a href="https:&#x2F;&#x2F;tapr.org&#x2F;product&#x2F;tapr-ticc&#x2F;" rel="nofollow">https:&#x2F;&#x2F;tapr.org&#x2F;product&#x2F;tapr-ticc&#x2F;</a><p>The annoying thing is there are few PC hardware combos that include the relevant PPS and frequency outputs to allow actual verification of your time in hardware. Software can show numbers but even on my Pi dashboard, the sub-20ns sync reported is between the system clock and the NIC PHC... but not currently accounting for the delay when the kernel reads the PHC over PCIe.<p>Pi <i>could</i> support PTM on its PCIe bus, that would be awesome. Overkill for 99.9% of its market though!
        • gsich5 minutes ago
          Intel I225 has support.
  • yardstick1 hour ago
    It’s been a decade or more since I last used NTP pool on my servers, have they fixed their quality issues with unreliable and misconfigured time sources? With 10k+ servers polling them, I saw enough issues to abandon it in favour of the likes of Google, Apple, Microsoft etc.<p>Issues included unresponsive servers, and servers handing out time that was days&#x2F;weeks&#x2F;months out of realtime.
  • lightswitch056 hours ago
    Great write up! I too run a stratum 1 and am part of the pool. Except, I have way less interesting hardware. One benefit on going cheap is having multiple stratum 1s then you can have some redundancy and have more interesting local peers to compare results against. I do see a more expensive setup in my future, though.<p>Anyways, I use NTPSec- which is the default these days on Debian when you apt-get install NTP. I’ve not tried Chrony, but I see it mentioned all the time.<p>I’m curious what others here are running and their thoughts on the pros and cons?<p>Edit: I forgot to add, one of my favorite parts of being in the pool is basically a free uptime monitor of my ISP. There have been multiple cases where I see a blip in my offset then a full blown ISP outage within 78 hours of the blip, then a return to my normal offsets after to outage has resolved.
    • geerlingguy5 hours ago
      In the NTP Pool community, I&#x27;ve seen a few people mention CenterClick NTP servers; I&#x27;m happy to do my own DIY builds (Pi or otherwise), but this looks like a nice option for SMBs or time-nuts who don&#x27;t want to manage their own hardware: <a href="https:&#x2F;&#x2F;centerclick.com&#x2F;ntp&#x2F;" rel="nofollow">https:&#x2F;&#x2F;centerclick.com&#x2F;ntp&#x2F;</a><p>There are also a lot of people who buy old Spectracom or other &#x27;enterprise&#x27; time servers since they can be had for low hundreds of dollars with some nice features like rubidium or OCXO oscillators, so they can hold over time much more accurately when GPS is down, or upstream NTP providers are unreachable.<p>And regarding chrony, it&#x27;s the default in many distros now, I like some of its built-in monitoring and clock adjustments, it just feels easier to manage if you want more than just base NTP features. It also supports hardware timestamping if you have network hardware that has it, which IIRC NTPd doesn&#x27;t? I haven&#x27;t messed with systemd-timesyncd or ntpd-rs, so can&#x27;t say much on them.
      • unsnap_biceps4 hours ago
        I have around a dozen CenterClick 270s at various locations and I think generally I have one lock up every 6 months or so. A PoE power cycle on the switch recovers it, so overall I&#x27;ve been really happy with them. I believe they&#x27;re a one-person shop.
        • cinntaile3 hours ago
          What do you use them for?
        • toomuchtodo3 hours ago
          Do you automate monitoring ntp health to power cycle the switch poe?
    • nahikoa5 hours ago
      Some of the cool kids are moving to <a href="https:&#x2F;&#x2F;github.com&#x2F;pendulum-project&#x2F;ntpd-rs" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;pendulum-project&#x2F;ntpd-rs</a>
    • Palomides5 hours ago
      I think most people run chrony or ntpd-rs; not part of the pool, but I&#x27;ve been using chrony for a while, and it seems like ntpd-rs&#x27;s support for PTP (which I &quot;need&quot;) is still under development
  • bityard4 hours ago
    I thought I remembered reading that they did not allow VPSes to be part of the pool because of the historical issues around VMs and timing, combined with the fact that the VPS operator could make host-level changes at any time that could impact the accuracy of the host. But now I&#x27;m not finding anything about that. Maybe they have enough monitoring and automation that running nptd on bare metal is not a strict requirement anymore?
    • walrus014 hours ago
      I don&#x27;t recall the details but one of the historical problems with VPSes is that people were trying to use things like openvz VPSes which are more like a glorified chroot&#x2F;jail, rather than something like a QEMU&#x2F;KVM VPS (or Xen PVH, or Xen HVM with virtualization extensions).<p>I still personally wouldn&#x27;t try to run an NTP server on a VM unless I also owned&#x2F;controlled the bare metal hardware the hypervisor is running on.
      • geerlingguy3 hours ago
        KVM allows chrony to adjust the clock, which give much better stability. With other VMs you can have a very jumpy clock and poor time service.<p>Some decent discussion on this topic (though with broken links to a separate NTP Pool discussion) here: <a href="https:&#x2F;&#x2F;a.osmarks.net&#x2F;content&#x2F;serverfault.com_en_all_2022-11&#x2F;questions&#x2F;812585&#x2F;things-to-consider-when-running-public-ntp-servers" rel="nofollow">https:&#x2F;&#x2F;a.osmarks.net&#x2F;content&#x2F;serverfault.com_en_all_2022-11...</a>
        • walrus013 hours ago
          I&#x27;m not sure if I would call answers from 2016 &#x27;recent&#x27; as KVM linux-on-linux has advanced significantly in the years since then, particularly with improvements focused on performance in something like a 6.1.x kernel in a theoretical scenario like a debian trixie host (proxmox 9.2 or whatever) and a trixie or fedora 44&#x2F;45 guest.
          • geerlingguy3 hours ago
            Touche, but the main point is KVM is fine for NTP timing. I don&#x27;t think I&#x27;d try to virtualize a PTP server, that still needs to be closer to the physical hardware.
  • tomassirio2 hours ago
    Hey Jeff, I learn a lot through your content.<p>Thanks a lot for sharing knowledge!
  • omoikane5 hours ago
    &gt; <a href="https:&#x2F;&#x2F;www.jeffgeerling.com&#x2F;blog&#x2F;2026&#x2F;i-joined-the-ntp-pool&#x2F;clock-rack-in-studio.jpg" rel="nofollow">https:&#x2F;&#x2F;www.jeffgeerling.com&#x2F;blog&#x2F;2026&#x2F;i-joined-the-ntp-pool...</a><p>I wonder why the wall clock on top is 10 seconds ahead of the bottom one?
    • geerlingguy4 hours ago
      Bottom one is NTP, updating I think once or twice a day. The top one is using WWVB radio time, but my studio is a little bit of a faraday cage, so to get it to sync I have to bring it near the front door and point the back toward the West overnight (STL to Ft. Collins).<p>I have a solution to that I&#x27;m working on—Pico W that grabs time from NTP then broadcasts about 1m with a tiny antenna so I can set my wall clock and watch (the one with the green band in the picture).
      • NooneAtAll51 hour ago
        how fast does the de-sync happen?
        • geerlingguy58 minutes ago
          The photo in the blog post was taken about a week after the last WWVB sync, so the clock runs approximately 10s fast&#x2F;week, or +16ppm, which is typical for cheap quartz crystals.<p>One annoying side effect of running faster is whenever the clock gets a correction, since the gears only run <i>forward</i>, the clock has to go all the way around almost 12 hours to get back to the corrected time after a sync.
    • undersuit4 hours ago
      Relativity. &#x2F;s
  • zodwick4 hours ago
    One of those posts which truly hits you in the face and calls you an idiot for how much you don&#x27;t know. Definitely a rabbit hole I&#x27;m going down this week. Thanks
  • jeffrallen5 hours ago
    PSA: The Swiss government has NTP servers on ntp{11,12,13}.metas.ch.<p>METAS is the Federal Institute of Metrology.
    • rmoriz4 hours ago
      Germany <a href="https:&#x2F;&#x2F;www.ptb.de&#x2F;cms&#x2F;ptb&#x2F;fachabteilungen&#x2F;abt9&#x2F;gruppe-95&#x2F;ref-952&#x2F;zeitsynchronisation-von-rechnern-mit-hilfe-des-network-time-protocol-ntp&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.ptb.de&#x2F;cms&#x2F;ptb&#x2F;fachabteilungen&#x2F;abt9&#x2F;gruppe-95&#x2F;re...</a><p>However using a pool is the better way IMHO.<p>Or go Stratum 1 via DCF77 <a href="https:&#x2F;&#x2F;zeit.ptb.de&#x2F;" rel="nofollow">https:&#x2F;&#x2F;zeit.ptb.de&#x2F;</a>
      • cf100clunk51 minutes ago
        Canadian government&#x27;s National Research Council: <a href="https:&#x2F;&#x2F;nrc.canada.ca&#x2F;en&#x2F;certifications-evaluations-standards&#x2F;canadas-official-time&#x2F;network-time-protocol-ntp" rel="nofollow">https:&#x2F;&#x2F;nrc.canada.ca&#x2F;en&#x2F;certifications-evaluations-standard...</a><p>NTP server: time.nrc.ca but use a pool instead
  • aaronax5 hours ago
    What speed did you choose in the pool dashboard? How many request per second are you serving, and do you have a goal for that?
    • geerlingguy5 hours ago
      I started at 25 Mbit to see how things would go, and found the Oxtrys VPS I have the public server running on could handle that without issue.<p>Over the past few weeks I bumped it up to 50, then 100, and now it&#x27;s at 250 Mbit. At 100 Mbit I was seeing 180-250 requests&#x2F;sec. At 250 (just switched earlier this morning) I&#x27;m seeing 500-600 requests&#x2F;sec.<p>I&#x27;m monitoring things on the server with chronytop[1], a tool I whipped up with Claude since I didn&#x27;t see any other concise dashboards and hated `watch`ing chronyc sources to get any kind of updated data.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;geerlingguy&#x2F;chronytop" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;geerlingguy&#x2F;chronytop</a>