Something I didn’t see mentioned was that this unequal memory access time also affects pcie I/O. If your thread on CPU A needs to get data in or out of a nic on CPU B, your throughput/latency will be impacted.<p>We have to explain this to customers of our software all the time, it’s something that’s easy to miss.
(CTO of Edera here)<p>Great point! We also try to factor that in as well.<p>Steven (the author) will cover that in part 2!
Same. The drop in performance can be surprisingly bad. 10Gbps becomes 5Gbps. 100Gbps becomes 20Gbps.
When building Edera (product from article), I also had the added problem of the virtual networking gap where I was bridging a 10Gbit NIC over a virtual interface, and I had weird performance bouncing between 3Gbit and the full 10Gbit. Luckily I had built networking drivers before and knew the complexities of it, and managed to profile it down to the virtual interface getting worst-case NUMA occasionally.<p>The part 2 is going to cover how we actually solved it, which involves every part of the system having knowledge. It's so easy to ignore but it has a massive impact on perf.
NUMA can cause really crappy performance. We deployed a Go based LLM gateway in Kubernetes deployed on a server with hundreds of CPU cores. We didn't explicitly set GOMAXPROCS so Go runtime scheduled goroutines over different CPUs and it constantly used 200% CPU and GC was causing latency spikes. Then we set GOMAXPROCS 8 and all performance issues went away. Until recently Kubernetes didn't work well with NUMA.
Is this on AMD? I wonder if it's all to do with NUMA or their CCD architecture etc (well these days Intel and everyone also does it to some extent).
Intel suffers just as much when NUMA enters the picture, even prior to CCD style architecture. That extra latency hop across to the other core to get at memory is absolutely crippling, especially in a hot loop. It requires very careful handling, while being this kind of invisible element (unless you know to look for it, nothing will draw your attention to it)
Hundreds of cores is likely two sockets and so you've got NUMA there.<p>Scaling to large core counts has a lot of gotchas.
There is one instance where the NUMA performance never disappoints: <a href="https://www.youtube.com/watch?v=Cqd1Gvq-RBY" rel="nofollow">https://www.youtube.com/watch?v=Cqd1Gvq-RBY</a>
NUMA is one of those amazing things that trip you up in all sorts of ways at unexpected times. The amazing "invisible" performance killler (invisible because unless you're already aware of NUMA, or remember to check, you won't know it's there potentially crippling you.)<p>It has been a source of routine conversations with customers and engineers of all kinds, and often one of those things you don't know about until too late.<p>I don't know if the kernel has improved this behaviour in the several years since last tested, but a coworker realised that the linux page-cache wasn't fully split by NUMA node. They were benchmarking mysql running it in each NUMA node, and noticed the second NUMA node was noticeably slower. Then discover after a reboot the second node was fast, and the first was slower. After a bit of thinking and tinkering they discovered that libmysql was ending up in the page cache in the same NUMA node as the benchmark client was run in first, so even though they were pinning the benchmark tool and mysql process to the NUMA node, the benchmark client was causing the OS to reach <i>across</i> the NUMA node to get at the page cached library.
I'm baffled by the fact that NUMA is still an issue in 2026. My impression is that this was all solved back in dotcom era already on those big SUNs. At least in HPC we solved this already in mid 2000s. Why is supposedly modern world still wasting time on this? Kernel these days exposes just about everything you would ever want to know about a system topology and every runtime should be making use of that information. If it does not, I cannot consider it ready for this century.
Because numa topology is an optimisation problem with a wide solution space, and that its configuration and setup depends on the amount of physical cpus and cores; how the RAM is connected to which lanes; and on and on it goes.<p>> If it does not, I cannot consider it ready for this century.<p>Mhmm.
Yeah, when you have tall servers this can be a really surprising factor. In some sense you could view this as an extension of processor caching behaviors, which also causes some memory accesses to be lower - just due to cache behaviors, not physical location. But in many cases, the same tools can be used to fight both "far" memory accesses and cache trashing, by using a thread-isolated architecture.<p>I have been dealing with the topic for a few years now and it was surprisingly hard to track down the bottlenecks to actual numbers. Some time ago I managed to find a good example to demonstrate the effect in a tangible way and wrote up an article about it. If the topic sounds interesting, you might enjoy <a href="https://sander.saares.eu/2025/03/31/structural-changes-for-48-throughput-in-a-rust-web-service/" rel="nofollow">https://sander.saares.eu/2025/03/31/structural-changes-for-4...</a> (Structural changes for +48-89% throughput in a Rust web service).
[dead]