CXL is going to be really interesting.<p>On the positive side, you can scale out memory quite a lot, fill up PCI slots, even have memory external to your chassis. Memory tiering has a lot of potential.<p>On the negative side, you've got latency costs to swallow up. You don't get distance from CPU for free (there's a reason the memory on your motherboard is as close as practical to the CPU)
<a href="https://www.nextplatform.com/2022/12/05/just-how-bad-is-cxl-memory-latency/" rel="nofollow">https://www.nextplatform.com/2022/12/05/just-how-bad-is-cxl-...</a>. CXL spec for 2.0 is at about 200ns of latency added to all calls to what is stored in memory, so when using it you've got to think carefully about how you approach using it, or you'll cripple yourself.<p>There's been work on the OS side around data locality, but CXL stuff hasn't been widely available, so there's an element of "Well, we'll have to see".<p>Azure has some interesting whitepapers out as they've been investigating ways to use CXL with VMs, <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2023/06/2023-CXL-DesignTradeoffs-IEEE-Micro.pdf" rel="nofollow">https://www.microsoft.com/en-us/research/wp-content/uploads/...</a>.
Yup, for best results you wouldn't just dump your existing pointer-chasing and linked-list data structures to CXL (like the Optane's transparent mode was, whatever it was called).<p>But CXL-backed memory can use your CPU caches as usual and the PCIe 5.0 lane throughput is still good, assuming that the CXL controller/DRAM side doesn't become a bottleneck. So you could design your engines and data structures to account for these tradeoffs. Like fetching/scanning columnar data structures, prefetching to hide latency etc. You <i>probably</i> don't want to have global shared locks and frequent atomic operations on CXL-backed shared memory (once that becomes possible in theory with CXL3.0).<p>Edit: I'll plug my own article here - if you've wondered whether there were actual large-scale commercial products that used Intel's Optane as intended then Oracle database took good advantage of it (both the Exadata and plain database engines). One use was to have low latency durable (local) commits on Optane:<p><a href="https://tanelpoder.com/posts/testing-oracles-use-of-optane-persistent-memory/" rel="nofollow">https://tanelpoder.com/posts/testing-oracles-use-of-optane-p...</a><p>VMware supports it as well, but using it as a simpler layer for tiered memory.
> You probably don't want to have global shared locks and frequent atomic operations on CXL-backed shared memory (once that becomes possible in theory with CXL3.0).<p>I'd bet contested locks spend more time in cache than most other lines of memory so in practice a global lock might not be too bad.
Yep agreed, for single-host with CXL scenarios. I wrote this comment thinking about a hypothetical future CXL3.x+ scenario with multi-host fabric coherence where one could in theory put locks and control structures that protect shared access to CXL memory pools into the same shared CXL memory (so, no need for coordination over regular network at least).
DBMSs have been managing storage with different access times for decades and it should be pretty easy to adapt an existing engine. Or you could use it as a gigantic swap space. No clue whether additional kernel patches would be required for that.
Huh, 200ns is less than I imagined; even if it is still almost 100x slower than regular RAM, it's still around 100x faster than NVMe storage.
I have never had to go deep into NUMA configuration personally but couldn't it be leveraged here?
What kind of motherboard, CPU, cables, switches, and end devices would I need to buy to have a CXL network?
CXL uses the PCIe physical layer, so you just need to buy hardware that understands the protocol, namely the CPU and the expansion boards. AMD Genoa (e.g. EPYC 9004) supports CXL 1.1 as well as Intel Saphire Rapids and all subsequent models do. For CXL memory expansion boards, you can get from Samsung or Marvell. I got a 128 GB model from Samsung with 25 GB/s read throughput.
CXL networking is still in the R&D stage.
The latency concern is completely overblown because CXL has cache coherence. The moment you do a second request to the same page it will be a cache hit.<p>I would be more worried about memory bandwidth. You can now add so much memory to your servers that it might take minutes to do a full in-memory table scan.
> Buy From One of the Regions Below
> Egypt<p>:-/<p>But, because I'm a good sport, I actually chased a couple of those links figuring that I could convert Egyptian Pound into USD but <<a href="https://www.sigma-computer.com/en/search?q=CXL%20R5X4" rel="nofollow">https://www.sigma-computer.com/en/search?q=CXL%20R5X4</a>> is "No results", and similar for the other ones that I could get to even load
Yeah I saw the same. I've been keeping an eye on the CXL world for ~5 years and so far it's 99% announcements, unveilings and great predictions. But the only CXL cards a consumer/small business can buy are some experimental-ish 64GB/128GB cards that you can actually buy today. Haven't seen any of my larger clients use it either. Both Intel Optane and DSSD storage efforts got discontinued after years of fanfare, from technical point of view, I hope that the same doesn't happen to CXL.
I think Meta has already rolled out some CXL hardware for memory tiering. Marvell, Samsung, Xconn and many others have built various memory chips and switching hardware up to CXL 3.0. All recent Intel and AMD CPUs support CXL.
That is pretty hilarious. I wonder what’s the reason behind this. Maybe they wanted plausible deniability in case someone tried to buy it (“oh the phone lines were down, you’ll have to go there to buy one”).
I think someone just forgot to delete an option somewhere and it "crept in", and it really isn't supposed to have a "buy" link at all at this point.
Ok, I rented a camel and went to the specified location, but there was nothing there but some scorpions and an asp. What gives?
CXL is a standard for compute and I/O extension over PCIe signaling which has been around for a few years, with a couple of available RAM boards (from SMART and others).<p>I think the main bridge chipsets come from Microchip (this one) and Montage.<p>This Gigabyte product is interesting since it’s a little lower end than most AXL solutions - so far AXL memory expansion has mostly appeared in esoteric racked designs like the particularly wild <a href="https://www.servethehome.com/cxl-paradigm-shift-asus-rs520qa-e13-rs8u-2u-4-node-amd-epyc-server-review/" rel="nofollow">https://www.servethehome.com/cxl-paradigm-shift-asus-rs520qa...</a> .
The "AI" marketing on this is positively silly (and a good reflection of how weird everything has gotten in this industry.)<p>Do like the card though, was waiting for someone to make an affordable version (or rather: this <i>looks</i> affordable, I hope it will be both that and actually obtainable. CXL was kinda locked away so far…)
I’m really looking forward to GPU-CXL integration.<p>"CXL-GPU: Pushing GPU Memory Boundaries with the Integration of CXL Technologies"
<a href="https://arxiv.org/abs/2506.15601" rel="nofollow">https://arxiv.org/abs/2506.15601</a>
My god - a CXL product! That's really surprising anything go that far.
I'd been expecting external CXL boxes, not internal stuff.
Assuming you have the requisite CPU and motherboard with this card, does the memory just appear as normal under Linux/Windows/whatever OS is installed? Or do you need to get special drivers or other particular software to make use of it?
As someone not well versed in GPU and CXL, would someone mind explaining the significance of this.
That is amazing. Most consumer boards will only have 32 or 64. To have 512 is great!
You havent seen the price of 128GB DDR5 RDIMMs, they are maybe $1300 each.<p>A lot of the initial use cases of CXL seem to be to use up lots of older DDR4 RDIMMs in newer systems to expand memory, eg cloud providers have a lot.
... and if you have the money, you can use 3 out of 4 PCIe5 slots for CXL expansion. So that could be 2TB DRAM + 1.5TB DRAM-over-CXL, all cache coherent thanks to CXL.mem.<p>I guess there are some use cases for this for local users, but I think the biggest wins could come from the CXL <i>shared</i> memory arrays in smaller clusters. So you could, for example, cache the entire build-side of a big hash join in the shared CXL memory and let all other nodes performing the join see the single shared dataset. Or build a "coherent global buffer cache" using CPU+PCI+CXL hardware, like Oracle Real Application Clusters has been doing with software+NICs for the last 30 years.<p>Edit: One example of the CXL shared memory pool devices is Samsung CMM-B. Still just an announcement, haven't seen it in the wild. So, CXL arrays might become something like the SAN arrays in the future - with direct loading to CPU cache (with cache coherence) and being byte-addressable.<p><a href="https://semiconductor.samsung.com/news-events/tech-blog/cxl-memory-module-box-cmm-b/" rel="nofollow">https://semiconductor.samsung.com/news-events/tech-blog/cxl-...</a>
Both of the supported motherboards support installation of 2TB of DRAM.
Why did something like this take so long to exist? I’ve always wanted swap or tmpfs available on old RAM I have lying around.
Such things have existed for quite a long time...<p>For example:<p><a href="https://en.wikipedia.org/wiki/I-RAM" rel="nofollow">https://en.wikipedia.org/wiki/I-RAM</a><p>(Not a unique thing, merely the first one I found).<p>And then there are the more exotic options, like the stuff that these folk used to make: <a href="https://en.wikipedia.org/wiki/Texas_Memory_Systems" rel="nofollow">https://en.wikipedia.org/wiki/Texas_Memory_Systems</a> - iirc - Eve Online used the RamSan product line (apparently starting in 2005: <a href="https://www.eveonline.com/news/view/a-history-of-eve-database-server-hardware" rel="nofollow">https://www.eveonline.com/news/view/a-history-of-eve-databas...</a> )
Yeah. I can't count how many times I've seen descriptions of northbridge links smelling like the author knows it's PCIe under the hood. I've also seen someone explaining that it can't be done on most CPUs unless all cache systems are turned off because (IO?)MMU don't allow caching of MMIO addresses outside DRAM range.<p>The technical explanations for the fact that you (boolean)can't have extra DRAM controllers on PCIe is increasingly sounding like market segmentation reasons than purely technical ones. x86 is a memory mapped I/O platform. Why we can't just have RAM sticks on RAM addresses.<p>The reverse of this works btw. NVMe drives can use Host Memory Buffer to cache reads and writes on system RAM - the feature that jammed and caught fire on recently rumored bad ntfs.sys incident in Windows 11.
I'd have rather a question why we had single (or already dual) core CPUs with dual-channel memory controller and now we have 16-core CPUs but still with only dual-channel RAM.
AMD EPYC has 12 channel, 24 on dual socket. AMD sell machines with 2 (consumer), 4 (threadripper), 6 (dense edge), 8 (threadripper pro) and 12 memory channels (EPYC high end). Next generation EPYC will have 16 channels. Roughly if you look at the AMD options, they give you 2 memory channels per 16 cores. CPUs tend to be somewhat limited in what bandwidth they can use, eg on Apple Silicon you cant actually consume all the memory bandwidth on the wider options just on the CPUs, its mainly useful for the GPU. DDR5 was double speed of DDR4, and speeds have been ramping up too, so there have been improvements there.
DDR1 and DDR2 were clocked 20x and 10x slower than DDR5. The CPU cores we have now are faster but not <i>that much</i> faster, and with the typical user having 8 or fewer performance cores 128 bits of memory width has stayed a good balance.<p>If you need a lot of memory bandwidth, workstation boards have DDR5 at 256-512 bits wide. Apple Silicon supports that range on Pro and Max, and Ultra is 1024.<p>(I'm using bits instead of channels because channels/subchannels can be 16 or 32 or 64 bits wide.)
Intel and AMD I'd reckon. Apple went wide with their busses.
Well, Each Channel needs a lot of pins. I don't <i>think</i> all 288/262 pins need to go to the CPU, but a large number of them do, I'd wager; The old LGA 1366 (Tri-Channel) and LGA 1151 (Dual Channel) are probably as close as we can get to a simple reference point [0].<p>Apple FBOW, based on a quick and sloppy count of a reballing jig [1], has something on the order of 2500-2700 balls on an M2 CPU.<p>I think AMD's FP11 'socket' (it's really just a standard ball grid array) pinout is something on the order of 2000-2100 balls and that gets you four 64 Bit DDR channels (I think Apple works a bit different and uses 16 bit channels, thus the 'channel count' for an M2 is higher.)<p>Which is a roundabout way of saying, AMD and Intel probably <i>can</i> match the bandwidth but to do so likely would require moving to soldered CPUS which would be a huge paradigm shift for all the existing boardmakers/etc.<p>[0] - They do have other tradeoffs; namely that 1151 has built in PCIE, on the other hand the link to the PCH is AFAIR a good bit thinner than the QPI link on the 1366.<p>[1] - <a href="https://www.masterliuonline.com/products/a2179-a1932-cpu-reballing-jig?VariantsId=10441" rel="nofollow">https://www.masterliuonline.com/products/a2179-a1932-cpu-reb...</a> . I counted ~55 rows along the top and ~48 rows on the side...
Completely agree, and this is a bit of a ramble...<p>I think part of might be that Apple recognized that integrated GPUs require a lot of bulk memory bandwidth. I noticed this with their tablet derivative cores having memory bandwidth that tended to scale with screen size but Samsung and Qualcomm didn't bother for ages. And it sucked doing high speed vision systems on their chips because of it.<p>For years Intel had been slowly beefing up the L2/L3/L4.<p>M1Max is somewhere between Nvidia 1080 and 1080TI in bulk bandwidth. The lowest end M chips aren't competitive, but near everything above that overlaps even the current gen NVIDA 4050+ offerings
Check out Strix Halo 395+ it’s got 8 memory channels up to 128 GB and 16 cores
[dead]
(S)ATA or PCI to DRAM adapters were widely available until NAND became cheaper per bit than DRAM, at which point the use for it kind of went away.<p>IIRC Intel even made a DRAM card that was drum-memory compatible.
RAM controllers are expensive enough that it's rarely worth pairing them with old RAM lying around.
For every gold rush, make and sell shovels.
I don't get it. The point of (ddrN) memory is latency. If its on the far side of pcie latency is much worse than the system memory. In what sense is this better than ssd on the far side of pcie?
It's only ~2x worse latency than main memory but 100x lower than SSD.
pcie devices can also do direct transfers to each other - if you have one of these and a gpu its relatively quick to move data between them without bouncing through main ram
The i in that logo seems like it’s hurting the A
I wonder whose controller they are using.<p>For a memory controller, that thing looks hot!
From the manual, it looks like a Microchip PM8712 [1].<p>[1] PDF Data sheet: <a href="https://ww1.microchip.com/downloads/aemDocuments/documents/DCS/ProductDocuments/Brochures/SMC-2100-Smart-Memory-Controllers-00005431.pdf" rel="nofollow">https://ww1.microchip.com/downloads/aemDocuments/documents/D...</a>
The Amiga approach resurrected.