The article features a truck that the chips get loaded onto. This quote from Tanenbaum immediately jumped into mind:<p>"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."
I was wondering how much financial value they put in one truck. Presumably the value density of this stuff is incredibly high (although not really to anyone else, so security probably isn't a huge issue). At what point do you split the shipment between multiple trucks to avoid huge losses if there's a crash?<p>When I worked on satellites we had some electronics which used a radiation hardened FPGA which cost something like $500k per chip. Carrying that around was nerve wracking.
Ha - I still remember when I worked on the IBEX-Lo Sensor, we had to hand carry it to Switzerland. One lucky member of our team got to fly first class with it because it wouldn't fit in a coach seat!
I once had to keep a $20M (technically; a million download codes for the same $20 movie for a promotion) thumbdrive in my home safe because it arrived too late to put in the safety deposit box like it was supposed to be.<p>I had to remind myself a few times the <i>practical</i> value to anyone was about $22.
They probably have insurance which means it doesn’t matter if it is all in one car or not. And even if they didn’t Samsung is big enough where a loss of a couple million dollars of components isn’t gonna bankrupt them.
It's treated the same way as any other generic shipment of cargo, really. It's <i>fine</i> throwing $20 million in one truck.<p>Security <i>is</i> an issue, though, these are treated as a high value load and typically would be followed by unmarked, plain clothes armed security guards tailing around the truck. (That's how Apple often does warehouse deliveries of trucks full of iPhones. It wouldn't surprise me if there was going to be some rolling lead soon following some trucks full of Duos...)
Almost 10 years ago at my university in Chile. My professor of Computer Networks invited a PhD Student that was working on the ALMA Observatory [0] doing some research on signal processing.<p>What stayed with me from that talk was that all the data that was captured from those massive antennas was packaged in boxes, loaded in a cargo van and travel 1.600 kilometers to Santiago to be uploaded into the systems.<p>[0] <a href="https://www.almaobservatory.org" rel="nofollow">https://www.almaobservatory.org</a>
Just hope nothing changes the latency is terrible.
I’ve often wondered about die thinning. I mean just looking at sd cards and micro sd etc, it was clear that there had to be a thinning step (or many steps) and it seemed amazing to me that such a complex step could be economically viable. And yet it clearly is. All a matter of being able to operate at scale I guess.<p>It’s not a step I’ve seen discussed very much, so it was kind of neat to see it discussed front and center in a lay article like this.
Planarization has always been part of the process: <a href="https://jeez-semicon.com/blog/Planarization-in-Semiconductor-Manufacturing-Complete-Guide/" rel="nofollow">https://jeez-semicon.com/blog/Planarization-in-Semiconductor...</a><p><a href="https://resources.pcb.cadence.com/blog/2023-the-planarization-process-for-semiconductor-manufacturing" rel="nofollow">https://resources.pcb.cadence.com/blog/2023-the-planarizatio...</a><p>Not dissimilar to lens or mirror making, the process of grinding to a precise flatness. The key step here is bonding it to a glass carrier, presumably because otherwise it gets too thin to support itself.<p>See also the "reduced kerf diamond wire saw": technology critical to cheap solar panels is the ability to saw the boule into thinner and thinner pieces with less waste at each cut, like a big ham slicer.
Yeah, the scale is kind of mind-bending. You're taking something that's already extremely fragile, grinding most of its thickness away, handling it through several more process steps, and somehow doing that cheaply enough for mass-market products.
The critical back end step for HBM is thermal compression bonding - where they actually compress 13 very thin die separated by solder balls. Cracking is a major issue.
Automation, automation, automation.<p>Zero chance you could do it if a human had to touch the die/chip directly during those steps.<p>But clean room + vacuum pick and place machine? A matter of fine tuning, and it should work just fine.
On a related note, I was reading yesterday that apparently the real bottleneck for Chinese production of AI accelerators is HBM production, not processors or ASML equipment.<p>The lack of ASML EUV machines certainly hurts, and pushing DUV so hard results in abysmal yields of good chips, but you can compensate by running more wafers or making smaller chips, and the net result is that Huawei's Ascend production volume is limited by CXMT's HBM capacity not processor dies.<p>The problem is that HBM manufacture requires many steps (die thinning, via drilling, plating, alignment) where the equipment used by everyone else (Samsung, SK Hynix, Micron) is also blocked by sanctions, so the Chinese are having to develop all of this themselves too, which they have, but yields are currently low, even when using shorter HBM stacks.
The only thing the West is achieving here is that sooner or later China will be able to compete on their own terms rather than ours. It may buy some time but the end result is very predictable.
As other comments pointed out, that was always China's plan, the value here is really that when the Chinese overlord annillate the world's industrial competition, they'll have the moral high ground of being out of self-reliance and hostility by western monopolists.<p>Edit:<p>And, that, my fellow keyboard high-minded thinker but powerless corp peons, was also part of the China's plan all along.
Eh. Industrial policy in a big country is always the output of politics between stakeholders, anthropomorphizing it to a "guy with a plan", especially an "overlord" looking to "annihilate" is probably getting it wrong.<p>A friend of a friend is married to a member of the national academy of sciences in China. I heard at dinner a few years ago that he had been lobbying for more onshoring of chip production for years before the Nvidia restrictions, and always heard "you're right, maybe next year we'll be able to fund that". Then the restrictions came in and everything changed.
China's plan to be self-sufficient in semiconductors was a failure before western sanctions, because all the Chinese chipmakers found it too easy to buy western chipmaking equipment or to fab through TSCM. No Chinese company wanted to buy from other Chinese companies. Nobody was incentivized to develop self-sufficiency. "Because the government wants it" was not enough incentive.<p>Western sanctions have succeeded in achieving China's self-sufficiency goals where their own government failed.
> [...] that was always China's plan, [...] they'll have the moral high ground<p>It's a possibility. It is also possible that they will steal yesterday's technology and still pretend to have moral high ground.<p>I do not wish China and Chinees wrong but most probably communism will work as it always do - by faking thing and bringing self-harm to their own citizens.
In free markets, everyone steals everything they can, and stops everyone else stealing from them as much as they can. This is normal and expected. The ideology says that it will even out and converge to a fair situation.
And by stealing you mean exchange? Or I somehow misunderstood what you consider a <i>market</i> to be - a place for pickpockets?
> In free markets, everyone steals everything they can, and stops everyone else stealing from them as much as they can. This is normal and expected. The ideology says that it will even out and converge to a fair situation.<p>This is all nonsense. Capitalism isn't an ideology, it's just a slight formalisation of what naturally arises. People are free to choose; they can own things; they can make agreements; they must honour the agreements or pay a penalty. Capitalism in no way is about stealing.
<p><pre><code> > Capitalism isn't an ideology
</code></pre>
your confusing trading/markets with capitalism; they aren’t the same thing, and it definitely has ideology(ies)<p><a href="https://www.britannica.com/money/capitalism" rel="nofollow">https://www.britannica.com/money/capitalism</a>
It’s the same in communism/socialism except people are prevented from stopping everyone else from stealing their stuff, in the name of the ‘greater good’, and if they try to get more than everyone else get guilt tripped into giving it up under the ‘to everyone what they need, from everyone what they can’.<p>Which incentivizes pretending you need a ton, and can give nothing. You think being useful at work in Capitalism is bad? Wait until you are accidentally useful in a communist/socialist economy.<p>There is usually less pretending under capitalism, but people are people, so it all the same stories still play out.<p>People are shitty, if given the chance and influenced in specific ways. Capitalism or not. We’re dealing with the current bullshit in the west because somehow people bought the line that everyone is a good person - and we totally can’t actually see or do anything if we see someone doing something bad! - and we need to always follow ‘the rules’ even when someone bad clearly isn’t.<p>Until we find our balls, good luck.
[dead]
Oef, still kicking the “communism” horse. It’s dead Jim, especially in China it’s dead and buried
Was it also the Chinese plan all along to annihilate their own birh rate?<p>Was the Great Leap Forward part of the plan too?
> Was it also the Chinese plan all along to annihilate their own birh rate?<p>No, that's generally what happens when countries stop being poor. Taiwan, Singapore, Korea, all have similar fertility rates.
Yes, they literally named it the ‘one child policy’. It was just overly successful.
What the West is doing is forcing China into a situation which is impossible to sustain when their economic dividend runs out. Especially considering these investments are fuelled by an extremely high level of debt.<p>And it’s not a given that they’ll catch up. I think it’s fair to say that their development of jet engines is on track to be obsolete before they’re competitive.<p>They may in principle have established the technologies to be self sustained but that’s only relevant and sustainable if they develop an economy based on domestic consumption rather than exports. Which they are struggling with as well.
> their economic dividend runs out<p>i dont think it will - they still have a chokehold on battery production (which implies dominance in the EV sector), and they also control the vast majority of all rare earths. The chinese gov't can subsidize chip research/development with these other sectors that are dominating.<p>The west is the one that need to watch out for their own economic dividend running out from the past century.
What makes you think they don't have a domestic market?<p>The Chinese domestic EV market is the largest in the world.<p>Chinese factories are full of Chinese robotics.<p>Companies like Huawei are struggling to meet the domestic demand for AI accelerators, while growing volume very fast year over year.<p>The Chinese domestic AI market is massive, but we just don't hear much about it outside of the models that are also sold here. Can you even name the models that ByteDance (China's Meta) makes?<p>As far as exports go, China is a low cost producer for many things. If the US doesn't want to buy high quality EV's costing 1/2 the price of a Tesla, then there are plenty of other countries happy to do so. The US is only 5% of the global population - there are lots of other consumers out there.
> They may in principle have established the technologies to be self sustained but that’s only relevant and sustainable if they develop an economy based on domestic consumption rather than exports. Which they are struggling with as well.<p>Is this really true? Can't they just keep exporting "forever"? What requires them to find a domestic audience? I can think of some other countries that have done quite well mostly without finding a domestic audience (eg, Switzerland comes to mind, and frankly Europe might largely be in the same bucket).
<a href="https://www.imf.org/en/publications/fandd/issues/2023/06/superpowers-are-forsaking-free-trade-ngaire-woods" rel="nofollow">https://www.imf.org/en/publications/fandd/issues/2023/06/sup...</a>
Europe (by which I assume you mean EU or possibly Euro area) has a fairly balanced balance of trade overall.<p>And a teeny-tiny country like Switzerland can't be usefully compared to something as big as China - the rest of the world can easily consume whatever the Swiss produce, but there are limits to how much Chinese can produce until consumption will have to shift to domestic consumption because the world doesn't have infinite ability to pay for its demand.<p>That said don't expect any big change anytime soon - there are very strong political incentives to keep the status quo with China focusing a bit too much on exports.
It baffles me how shortsighted the policymaking here is. Like, what did they expect to happen?
They expect to retire with a decent amount of money before things go down hill.
They probably weren't expecting AI foundation models / model research to be quite as fungible as they are
Yes.<p>Over the last 20 years the economy has become dysfunctional. It no longer really resembles a free market; monopolies have established barriers to entry everywhere. And the biggest investors are awash with helicopter money that's been doled out for favors by the political class.<p>So those investors have tons of cash to burn and surprisingly few opportunities. Even within Silicon Valley/VC there are surprisingly few who seem to really understand the fundamental economics of software. Or perhaps those economics just aren't that important when you have billions of dollars on hand and cash is obviously not going to get you a return. Any whisper of possible exponential growth is worth throwing money at. Crypto? Why not. AI? Why not. Datacenters? Why not. Tulips? Why not.<p>This is by all definitions an empire in decline. Everything is broken or fake. Everyone is afraid to do what needs to be done. Power forbids it. So we're all just waiting for the other shoe to drop. Our secret police aren't as bad as the late stage USSR's yet, but hold Uncle Sam's beer...<p>(That's not a recommendation to try and time the nadir, by the way, as it could easily be 50 years away.)
Datacenters and AI obviously work and come with a credible value proposition and path to profitability.<p>Comparing this to crypto or turnips makes me doubt the merit of any other opinions your comment offers.
The initial growth of the internet was a boom bust cycle, and as you see the internet is still here. This is more about the economic paradigm that is being used to grow a technology versus the economic paradigm the technologies rate of growth can support over time.<p>Just to give some rough numbers in the past 5 years the amount of GPU compute installed in FLOPS is somewhere over 5 times all the CPU flops that have ever existed.<p>This has nothing to do with AI being good or bad or being able to produce things and economic value. It is by far the largest and fastest growth of any technology ever and we have zero clue about the economic stability of this grand experiment we're performing.
The internet is a much more appropriate comparison than crypto or tulips, which bring no or negligible value.<p>Some differences to the market at that time seems to be that during the dotcom bubble, many of the companies had little to no revenue.<p>Leading AI labs already generate enormous revenue. The investments into capacity are needed to address the current demand.<p>The situation seems somewhat less speculative.<p>That said, I cannot predict how AI capabilities will develop and how demand will respond.<p>Should capabilities plateau hard and soon, maybe the demand will not be there for the compute investments.<p>If it does not, and instead AI applications in robotics, science, and self driving expand, chances seem reasonably good that the demand will be there, no?<p>And as for the economic stability, much of the investment comes from existing giants like Microsoft, Alphabet, Amazon, and Meta, who have the necessary cash flow.<p>These companies are less likely to collapse than some of the ones during the dotcom bubble.
Eh, I'd say it's closer to something like internet + tulips.<p>It's the total amount of money in the economy that's been invested toward a potential outcome. AI represents the largest amount of money, and largest fractional part of the economy invested ever.<p>Because of this AI could be the biggest economic boon ever, yet still not recover the full amount invested. This will have deep economic impacts that affect everyone and everything. At this point AI must achieve all its stated economic impacts or there will still be a huge economic crash that kills off any company that is over invested and cannot make a profit.<p>Worse, the many of the perceived economic impacts of AI are not for humans like you or I, but the huge companies you listed. Even if they economically win, everyone else made out of meat could still lose.<p>They say history doesn't repeat, but it does rhyme. This, at least to me, sounds like a mixtape of "internet" + "tulips" + "1920s financial world leading to global political instability".<p>Every potential outcome I see occurring pushes us closer to further instability, even if the economics on it work out on paper.
> At this point AI must achieve all its stated economic impacts or there will still be a huge economic crash<p>I had a look at the numbers.<p>The investment into data centers is estimated around $800B/year currently.<p>OpenAI + Anthropic combined had ARR of >$100B in July.<p>Global labor income is around $65T/year, the US GDP $32T, the global one $126T.<p>Global software spending: $1.4T/year.<p>I am no financial analyst, but I don't think all the stated AI impacts have to be met just to recover the investments.<p>A 1% productivity gain on global labor income represents $660B/year.
At 5% we would look at $3.3T/year.<p>They don't need to cure cancer to justify the investment, even though Amodei hopes to cure most major disease in the next 5-10 years.
It started way before the last 20 years, the past 40 years have created the conditions for this to happen.<p>Overfinancialisation of the economy while removing safeguards to keep markets functional, like anti-trust enforcement, are the main forces behind this erosion. Through finance the focus of companies is completely shifted away from producing good products and services, and into how to extract more paper wealth from existing structures to the detriment of products.<p>Not enforcing anti-trust and letting behemoths to form which cannot be competed against since with their amount of capital they can either buy their competition outright or just price dump for long enough to make competition non-viable.<p>It's the failure of neoliberalism, and that agenda has been pushed into Western countries since the 1980s-1990s, it made enormous wealth for the few at the top while eroding whole societies, economically and socially, it's all dysfunctional.
I feel like my point is being over extrapolated a bit. All I was saying is that people in 2020 believed AI companies had much more of a moat then they actually do - that other companies would struggle to obtain enough data (y’know, downloading the internet to train models wasn’t legal back then) or resources to compete
I replied to the extrapolation, not to your comment directly, I think your point might be more fitting under that comment than mine, I just drilled down from the points above mine.
they expect the Chinese to bend their knee and beg for the sweet, sweet chip
they were sold an idea the AGI is right around the corner so even a slight delay might be critical to secure the "win"
> It baffles me how shortsighted the policymaking here is. Like, what did they expect to happen?<p>As someone who is myself 'old af' by tech standards (I'll be 53 very soon) the main problem is that the policymakers are all 70+ years old.<p>Their concept of what China even is is forever stuck in pre-2004 thinking from back when they still had significant neuroplasticity.<p>Vote these fucking fossils out, kids, before it is entirely too late.<p>(And, yeah, before someone inevitably brings it up, some exceptions-that-prove-the-rule older policymakers/politicians do avoid this trap of getting stuck in the past, but most don't)
if you sell them ASML equipment your market gets destroyed tomorrow instead of 10 years later so what options do they have?
No longer strangling their own economy would be a good start. Take a look at the magnificance that was the ESRS. A 300 A4 pages long list of yearly reporting duties. And the best thing is: Just because they're no longer nicely summarized in one place, doesn't mean the underlying duties went away.
Sanctions can absolutely create the incentive to build a domestic supply chain, but incentive doesn't automatically translate into catching up on the same timeline
Well Soviet Union tried to be self-sufficient, but the truth is such strategy loses to global free trade. Whether we want it or not, global trade is what keeps the world from WW3 for now, not the nuclear MAD.
The Soviet Union never was the manufacturing power behind a very significant fraction of the worlds products.
Few countries rely more economically on global trade than China though.
You understand that the US has alienated almost all of its trading partners and is aggressively attempting to kill free global trade, right?<p>Whereas China is developing a powerful network of rapidly developing trading partners like Africa and the Middle East?
They always had the option to do that
people have been claiming Nvidia will get competition too for years but you guys seem to be forgetting that the current leaders are not asleep at the wheel in the meantime.
[dead]
When China eventually catches up on conventional lithography, the goalposts will shift to silicon photonics and optical logic, a transition DARPA laid the publicly visible groundwork for decades ago through programs like POEM and PIPES while keeping the mature implementations strictly classified until needed.
I think eventually The west will want to kneecap China, so I guess this is the best time to do so.
Tokens per second is almost entirely memory bandwidth at inference time, training obviously needs more compute but you can add more chips for that.
Not quite, it's got quite a bit more complicated with agentic use cases.<p>Prefill (input tokens) is heavily compute bound. And the ratio of input to output continues to rise, as typically in agentic sessions you have a few tokens output for a tool call and (many) thousands of input from the tool result.<p>Then you have cached input tokens, which is a totally different issue, system RAM or NVMe bound.<p>Obviously output tokens is VRAM memory bandwidth bound, but this is less and less of the bottleneck these days for overall agentic speed.
This is why I was careful to specify tokens per second not time to first token which is the prefill step you’re talking about. Clearly to run these models well you need both but as I said adding more chips or compute units can give you more latency where as overall memory bandwidth (throughput) is limited by access to fast memory.
I can easily use close to 100 million input tokens a day. A few million output tokens but at the end of the day maybe a thousand or so lines of code get written.
According to SemiAnalysis, both inference and post-training (RLVR) is mostly memory bandwidth bound. Only pre-training is compute bound, but it now only takes a small share of overall data center capacity.<p><a href="https://x.com/EugeneNg/status/2099315982959616369" rel="nofollow">https://x.com/EugeneNg/status/2099315982959616369</a>
[dead]
China has had a very strong incentive to recreate lithography and fab equipment for years, but a lot of the specialized HBM packaging chain was probably much lower priority until relatively recently
Huawei uses their own non-standard HBM called HiZQ probably not produced by CXMT.
China should invest in an analog inference chip. It's a hail mary but why not.
The two challenges there are firstly - that analog design has been a separate electrical engineering school for most of a century, so there are few who could design it - and secondly - that every single chip will have subtle variations in its computations, necessitating some sort of model finetuning per chip. Possibly the chip could be characterised at the factory, and ship with the characterisation data burned into a controller rom or something, but if that doesn’t pan out the whole thing is likely a non-starter.<p>If it could be made to work, you could run a fable-grade model in tens of watts.
Or we treat them like humans and make each one do job interviews to work out what tasks they’re suited for.
This made me smile. A true test of human-like intelligence. Maybe they could take some entry level jobs (“I can make a nice Wordpress site”) and build a CV?
It's the hidden sociopath that's able to mirror a functional entity that I'd be a bit worried about in these cases. Damn thing would make CEO quick.
awesome explanation. never thought about the complexity of doing analog and having precision issues because it's not discrete anymore and repeatable
They can probably afford to do both.
That would be like skipping land line phones for mobile...
Or on DRAM on die compute
I'd assume that they have - but will keep mum 'till they have a major breakthrough or large-scale operational deployment to announce.
The "abysmal" yield using DUV is an overblown statement by western commentators who don't look closer at the development. Certainly yield is worse than with EUV, but after multiple iterations of development, yield has become pretty good, within economically acceptable bounds, still making scaling possible. Volume is still ramping up. A lot of capacity will come online in 2027. The removal of western middlemen such as Mediatek actually improved the economics. Further yield improvements are still coming. Volume is already so high that domestically made phones and NPUs are making a real impact, yet are also selling like hot cakes.
It depends on what you are trying to make, but there are workarounds for most things, and it becomes more of a cost and volume issue than a showstopper.<p>If your yield is low and you still want the volume, then run more wafers, but then you need access to more DUV machines and more wafers. If your defect density is too high, then design smaller chips with a higher chance of avoiding defects, which is what Hauwei are doing - split processor into multiple chiplets (NVidia do this too, to raise yields and reduce cost).<p>HBM3E usually has 5-6000 vias, but doing this with multi-pattern DUV without defects is tough, so CXMT currently drop that to 3000, at a cost of some loss of thermal and voltage stability.<p>HBM:processor production ratios aren't what Huawei would like, so they mitigate this at system level by putting an optical memory bus on the GPU chip and sharing memory across the system.<p>Not everything is a huge LLM - smaller models like recommendation systems don't have so many parameters and need so much memory, so why waste HBM on them? ByteDance use their SeedChip acelerator for this, currently made by TSMC (using an older sanctions-approved 28mm process), which instead etches dense RRAM on die beside the processor.<p>There is also a time dynamic to this, with China still using pre-sanctions equipment and chips as their domestic alternatives ramp up to replace them. One interesting part of this is HBM ... HBM is very demanding to make, and everyone had problems with it, with initially only SK Hynix being successful. Samsung and Micron took a year or so to catch up, and during this time Samsung had made a ton of HBM that didn't meet NVidia's specifications, so ended up, pre-sanctions, selling it to China, where it has acted as a stockpile to carry them over as CXMT's domestic capacity ramps up.<p>China seems to be doing fine. Of course they would like sanctions lifted, moreso for HBM than anything else, but all that sanctions have really achieved is accelerating their semiconductor independence. They are still building 1T+ SOTA models, standing up 100K clusters of domestic AI accelerators, etc, etc.
I read HBM yields are 25-30% (vs 80-90%) making them 3 to 5 times as expensive. They are 4-5 years behind, that probably means 1-2 in Chinese time.
What's the main blocker (other than the current inflated cost) for using HBM instead of DRAM as the primary memory for consumer electronics?
It's not that there's a blocker. It's that it takes roughly 3x the manufacturing capacity to produce an HBM package at the same storage capacity as DRAM. We are sacrificing total bytes for bandwidth.
I don't fully understand the source of the "total bytes" constraint, but a major factor may be because HBM4 / HBM4E can only make use of the footprint directly above the processor/logic die (or in direct vicinity of its interconnect), while traditional DRAM can be placed further away where there's lots of real estate on the motherboard.<p>I gather a practical max ceiling today is a stack of 16 chips in height yielding 64GB?<p>These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5. That's what gives them their order-of-magnitude bandwidth speedup. But even though they technically pack in more capacity per <i>square millimeter of motherboard</i>, I gather they take up more space than older technologies once you account for the vias and interconnects to route all those signals.
Thanks for that, just went down an interesting rabbit hole. Many of us were hoping this re-tooling would eventually trickle some fast RAM down to DRAM-exhausted PCs, but given it would require a rearchitecture of the motherboard it's unlikely.
HBM4 has over 2048 signals to the processor’s PHY with tight signal integrity requirements that require the HBM stack to be < 0.5 mm from the processor die. That’s why HBM integration is done with interposers (soldered on the package). So, it’d be the CPU package that integrates it. Motherboard is too far away.
HBM also trades bandwidth for latency, and your regular computing is much more sensitive to latency than bandwidth.
How so? FEOL is pretty much the same, BEOL is almost the same save TSVs, the packaging tech is different and more advanced, but not exactly 1:1 comparable. Do TSVs really occupy 3x the area of DDR IO's?
See remark on the slide 11: <a href="https://www.servethehome.com/micron-evolving-memory-architectures-for-ai-at-hot-chips-2026/" rel="nofollow">https://www.servethehome.com/micron-evolving-memory-architec...</a> That presentation is by Micron.
It doesn't really say that it needs to use 3x more area, but that 3x more gets consumed due to "advanced packaging and manfuacturing complexity". Which doesn't properly explain why it consumes 3x more and could simply mean they have a bad yield and 2/3 produced is garbage.
You make HBM by stacking a whole bunch of dies on top of each other. The signals from the upper dies need to pass through vias in the lower dies to get out - taking up valuable die space in a way which simply isn't needed with regular DDR. Similarly, HBM has a <i>far</i> wider bus, so each individual die has, say, 16 banks of depth 32, rather than 4 banks of depth 128. That's more control area needed per byte of memory.<p>Those two combined already result in a huge reduction in bytes per mm2, so with the same wafer processing capacity you're producing far less byte of memory. Add to that a complicated chain of HBM-specific packaging steps, and you're now <i>also</i> losing a decent bunch of perfectly-fine dies because rather than putting it into DDR you tried making a HBM sandwich and screwed up.<p>Even if the memory cells are the same and have an absolutely identical yield, HBM will <i>always</i> end up having a significantly lower output. That's just the cost of stacking, but some people are willing to pay the per-gigabyte price penalty in return for the higher bandwidth.
> 2/3 produced is garbage.<p>This might not be far off the mark. You are irreversibly linking the fates of these devices after a certain stage of manufacturing. If something goes wrong at final packaging time, you lose all dies instead of one.
Yea, that's the question. Yield situation can improve. Area overhead would not improve short of a completely new and incompatible tech.
Classic DRAM stacks up to four wafers on top of each other and then is packaged with BGAs. The manufacturer can check the DRAM chips independently.<p>Soldering the DRAM onto a PCB is such a reliable process that there is almost zero risk of defects and even if a defect occurs the damage is limited. If the DRAM is soldered onto a DIMM the risk of a defect on the non memory hardware is non-existent. If the DRAM is soldered straight onto an SBC or GPU, then the DRAM can be removed to save the precious SoC or GPU chips.<p>Meanwhile HBM is the ultimate nightmare scenario. You stack up to 16 DRAM wafers on top of each other. One defect and the whole stack is worthless and that was actually the easy part.<p>In stage two things get even worse. You now have your accelerator chip and you must place the HBM on that chip. E.g. Blackwell GB300 has eight HBM stacks and the accelerator chip has a bigger area than the HBM. You must get the packaging right eight times in a row or you have wasted not only the DRAM silicon, but also the accelerator silicon because HBM cannot be removed and defects are permanent.<p>The issue here isn't just the yield of the HBM (which is obviously lower if you have taller stacks) but rather the yield of the combined HBM-based product, which is why doesn't make sense to say it needs more area but it is completely correct to say that HBM leads to more silicon being consumed. Hence it doesn't make sense to talk about yield of the HBM itself, because it is always part of an integrated product.
direct link: <a href="https://www.servethehome.com/micron-evolving-memory-architectures-for-ai-slide-11/" rel="nofollow">https://www.servethehome.com/micron-evolving-memory-architec...</a>
3x is a reasonable figure. They are not literally that large, though.
>> What's the main blocker (other than the current inflated cost) for using HBM instead of DRAM as the primary memory for consumer electronics?<p>HBM is meant to be integrated into the same package as the CPU, so no more DIMM sockets. It also has higher latency apparently.
One of the comments mentioned that they have a bus size of 2048, which may be why the latency is higher.<p>> These chips have a massive bus size of 2048 bits, instead of the 64 or 128 bits (dual channel) used by DDR5.
Why would you want/need to?<p>The advantage of HBM over regular non-stacked DRAM is memory bandwidth, which also requires a super-wide memory bus - 2048 bits wide for HBM4. Compare that to the 128 bit wide bus of a modern CPU.<p>So to take advantage of it on the desktop, or anywhere else, you need that 2048 bit wide bus, and a processor capable of consuming 2-3 TB of data per second!<p>These are not normal requirements, other than for a GPU.
> Compare that to the 128 bit wide bus of a modern CPU.<p>Or 256-512 bits on medium to high end consumer CPUs if you're apple.<p>At least DDR6 is probably widening things 50%.
Intel and IBM have already done this almost this with their wide eDRAM caches.<p>The advantage in going wide is transferring cache lines rapidly, not the CPU bus interface.
SIMD (and especially the modern matrix extensions) can use as much bandwidth you can throw at it.
Right but if HBM memory is all that people want to produce, then building a CPU which can use it use it would be useful on it's own merits.<p>But in reality we also already have unified memory architecture systems, integrated graphics etc.
I think people might prefer the lower idle power consumption from lpddr over the better bandwidth in hbm in battery powered stuff. That said right now the price is definitely preventing us from finding out.
Mainly bus width. Afaik HBM is like 1024 bits vs DDRs 64 so you need lots of transfers in parallel to saturate the bus, and CPUs kinda want 64 bytes of data as that's the size of a cache line ASAP. So you need a ton of in flight transfers which isn't a thing CPUs provide, maybe multicore workloads.<p>Buy the way you win with CPUs is with latency, and not bandwidth, which is why Apple M series actually uses DDR with lower latency because of the stacking.
HBM isn't dramatically better in every dimension. You get huge bandwidth and good energy efficiency per bit transferred, but not necessarily a meaningful latency improvement and capacity expansion becomes tied to the package
Even if the cost were the same of the RAM itself, you'd need much bigger and more expensive CPU to deal with it.<p>Running 17 chrome tabs doesn't benefit at all from that HBM and all the additional hardware+software complexities that come with it. You want a specialized coprocessor to handle specialized workloads. The GPU exists separately from the CPU for a reason.
HBM is a stack of DRAMs, so there is no “instead”.
In one sense, nothing, in another, everything. It is DRAM, but the bandwidth requirements mean it’s paired to a processor, i.e. no DIMMs. Not 100% sure but things like MacBooks and the Framework tower, where you have fixed RAM for the device lifetime, have ~0 tradeoff.
Nothing except CPU manufacturer choices. Mac laptops use it and they're consumer products.<p>People will have to get used to buying a fixed amount of RAM with their CPU but thats unlikely to be a problem.
Apple's "unified memory" marketing is so strong that even tech literate people seem to have this misconception!<p>They have managed to pull this sort of thing off many many times. <a href="https://en.wikipedia.org/wiki/Reality_distortion_field" rel="nofollow">https://en.wikipedia.org/wiki/Reality_distortion_field</a>
Yup, the only reason Macs have higher memory bandwidth is because they use more memory channels, which gives them a wider bus. Both Intel and AMD only allow more than dual channel memory on server class processors these days.
The “Apple only does X better because they do Y” thing has been a meme for ages. I remember dismissals like “PowerPC is only faster at math because it has more integer units” or something along those lines, and thinking, uh, isn’t that a good thing?
There's a qualitative difference between "they're doing a different thing" and "they're doing the same thing, tuned differently". GP is saying that this is a case of "they just tuned it differently".<p>This distinction doesn't change what the performance numbers look like today, but it does inform what changes would be necessary for those numbers to look different tomorrow. E.g. Apple Silicon isn't fundamentally orders of magnitude more efficient than x86, they just used smaller features. Newer Intel and AMD chips made on equivalent processes _also_ get similar efficiency gains.
There are AMD and Intel devices on similar process (not talking about A20Pro or M6 which are set to ship later this week), and they do not get the same gains.<p>And honestly, they have historically had different markets.<p>When the design is for only one customer, you don't need to generalize things, and those things you generalize to give different customers different options has costs.<p>AMD will soon be a larger customer for TSMC than Apple (NVIDIA is already there) so Apple's pre-booking new processes is likely to be gone in the near future.
The point is that "unified architecture" is a buzzword unrelated to what actually makes things fast.
Depends on the expense trade off.<p>If I get 10% more performance for 50% more cost it really depends on one's needs, for example.
"You're only better than me at sports because you practice more and try harder!"
I am just saying it's not magic and x86 is capable of doing the same. Quad channel memory used to be more common in consumer hardware but now it looks like you don't even have the option anymore for desktops. AMD's Strix Halo was the first sign to reversing that (it has quad channel memory!) and hopefully we see more of that in the future.
Threadripper have that extra bandwidth and M5 still is faster
Except strix halo.
The memory bandwidth is a small thing compared to the massive win you get by not having to move data between two memory pools at all.
Depends on your workload. And AMD has supported unified memory long before Apple Silicon existed anyway.
Often you don't move memory around as a programmer, but that's exactly what happens in the background.<p>It's the address space that's unified, not always the physical hardware.<p>The data movement (when needed) is handled transparently in the background by page faults and other tricks.
Having unified memory is a real advantage though, it's not a reality distortion.
It isn't really, as long as you don't care about power consumption, physical constraints and money. Basically desktops <2025 (and hopefully >2027).<p>DDR is optimized for latency and stability at the cost of bandwidth whilst GDDR is optimized for bandwidth at the cost of latency and stability. GDDR is pushed so hard these days that a small percentage of errors is expected and corrected because this is still faster than running it slower but more accurate.<p>GDDR7 often has 10-20x the total bandwidth but 3x the latency of DDR5. Graphical workloads want as much bandwidth as possible but care relatively little for latency. Conversely, applications love low latency but don't really see any performance benefit from higher bandwidth.<p>So basicallyt you have workloads that are diametrically opposed and running unified memory forces you to compromise.
The price is that you essentially glue CPU and GPU together, which limits total compute, from a size and thermal perspective.<p>This is really not a limit because of unified memory -- in principle, PCIe GPUs could read/write main memory without the CPU. But it's a limit for /fast/ unified memory, because fast means close.<p>So unified memory is great as long as the integrated GPU is strong enough. Then it has two advantages:
a) probably faster transfer CPU<->GPU (but that's an implementation choice for the non-unified case
b) If you either need a lot of memory for the CPU or the GPU, but not for both at the same time, you pay for memory only once.
It is a real advantage.<p>The reality distortion is that people seem to believe it's HBM, or somehow it gives you extraordinary amounts of vram. Neither are really true.
Agreed. That's why it's a good thing all computers made in the past 15 years have unified memory, not just macs.<p><a href="https://x.com/Lina_Hoshino/status/1820947147312820497" rel="nofollow">https://x.com/Lina_Hoshino/status/1820947147312820497</a>
In theory perhaps but the benefit is not as relevant with a weak iGPU. In practice all PC's with performance ambitions had a dGPU until Strix Halo and Panther Lake.
Is that Marcan's vtuber persona (Asahi Lina) that changed name since Marcan doesn't work on Asahi Linux anymore?
No I just failed to check before I posted, I thought it was HBM. I don't have any interest in Apple hardware so wasn't properly informed, and have been duly crucified.
MacBooks use regular soldered lpddr5(x) RAM. Same RAM as every other laptop manufacturer, they just use more lanes to achieve a higher bandwidth.
> Mac laptops use it and they're consumer products<p>No, Mac laptops use LPDDR, currently LPDDR5X.
The normal non-tech-savvy person already does this. They simply don't know that their ram is upgradeable or something else breaks first, before having to touch ram.
pretty sure its lpddr not hbm.
Will consumers actually get to purchase it though? Or are they just doubling it for frontier AI labs?
What does it matter?<p>All supply is supply, despite what certain folks seem to believe.
I have read an interview with Acer's CEO claiming that the shortage is already over.<p>He alleged memory maker's public announcements of the shortage lasting throughout the end of 2027 are attempts to covertly signal the others on pricing strategy.<p>With additional production capacity ramping up this year and next, this seems somewhat plausible.<p>If manufacturer's inventories are filling since they advanced orders when prices were rising, there could be an oversupply of memory when they don't place new orders at the higher current price.<p>This could quickly result in a price war while capacity expansion and yield improvements create irreversible oversupply.
As far as I know, the only place HBM is used in consumer products is GPUs?
A shame that this should if anything, lead to consumer DRAM prices getting even worse.
Right now, every company that can is ramping up their RAM production as much as humanly possible. There is such outrageous <i>and growing</i> demand, that it's like printing money if you can manufacture more RAM chips. And that massive financial incentive is driving huge investment in manufacturing capacity.<p>But someday, that demand will stop rising.<p>At some point, most people will have enough RAM. And they just won't need to buy more.<p>The inventories of RAM will become overfull, in warehouses and stores. And suddenly the manufacturers, middlemen, and even retailers will all find the price they can sell at keeps dropping.<p>But you can't shut down a huge manufacturing facility overnight. You can't shut down a years-in-the-making retail distribution system overnight, either. You can't even shut down a facility that is halfway finished with construction. They'll keep producing and selling RAM at lower and lower profit margins, until the day comes they finally accept they're losing money with each sale.<p>And when that happens, we'll all have more RAM than we know what to do with. It'll be cheaper than we have all ever experienced in our lifetimes.<p>I don't expect this to happen soon. But demand cannot keep exponentially rising <i>forever</i>. There will come a point, probably in the next 5 years, when demand at least levels off, even as production capacity is still rising.<p>And it will be interesting to see what's possible at that point. We'll be able to create software that is not doable right now, because it will no longer be constrained by memory. And even cheap laptops will massive amounts.
When you commoditize intelligence, you dramatically lower the cost to create. The amount of electronics being created is... incredible. We've really seem to have hit the asymtote part of the growth and i don't really see it coming back.<p>Everything is moving to edge compute. From smart rings and watches, to robotics and drones, iphones and humanoids, and no end in sight to the scaling improvements in AI... It does seem that you can continue to get more emergent behavior the more compute you expend in pre/ post training. There's no reason we can't scale up to a million data centers in space, as long as you get dramatically improved intellegence.
Yeah, that's kind of the beauty of capitalism. It can lead to short-term shortages but never structural ones. Any demand gets immediately fulfilled.<p>I grew up in a country where there was a structural shortage of say, shoes and cars. You didn't really even get to choose which shoe you wanted to get, when there was a new shipment, folks went to the "department store" and got whatever size they could get their hands on and later trading it for the right one. To get any car, you would submit an application and 5-8 years down the line you would get a letter that your car would now be ready to be picked up, in a random color.
The AI boom is turning into a massive race for HBM capacity.
tl;dr; This is existing production being redirected to HBM. Which means consumer market is going to get a lot worse.<p>And the worst thing is that is the best possible strategy for them. It's essentially win-win for everyone but consumers.<p>- Buyer (AI) has crazy money, so will pay whatever<p>- Seller doesn't have to build anything new, as the buyer is willing to pay whatever<p>- Generate ridiculous profits from crazy money<p>- No oversupply risk in case of reversal<p>- Return from producing HBM to DDR5 in a single quarter if reversal does happen.<p>Hold on to your existing hardware, people, and be on lookout for your local deals.
The solution would seem to be preventing that "Buyer (AI) has crazy money". They already have borrowed billions and are only generating losses so the money should run out. It just needs to be made sure that nobody (taxpayers) will rescue them.
This should be the top comment.<p>It absolutely is not a great situation at all. This won't make things better.
Will that be enough for AI's hunger?
During the Covid global chip shortage Intel announced new factories to address the production bottleneck, before the factories even started to be constructed, the shortage was long over and the projects were eventually canceled.
It will be just in time for when AI will run very well on consumer hardware and the need for data centers will collapse.
There will be demand for both
What if the AI companies start selling the hardware with their models?<p>I.e., you get a locked down device with access to their AI and maybe a way to run third party apps. Of course the developers of those apps will have to pay a percentage of their revenue.<p>Sound familiar?
That's never going to happen. By the time you can run current frontier models on your $10k desktop the frontier will have massively advanced and people will want those models instead.
I’m not so sure about that. Already AI vendors are back to cutting prices to try and keep customers from cutting back on their usage. My own employer is working hard at pivoting to much smaller fine-tuned models for established use cases, and seeing model performance improvement in addition to large inference cost reductions. Being able to run them locally hasn’t exactly been a disaster for devex, either.<p>It may turn out that demand for SOTA frontier models isn’t so limitless after all.
This + your ROI in $10k device will be always lower than busy datacenter, its literally math. They sell free compute to others when you dont use it, you will never sell at that level or even you magically sell home compute, you will not compete at price
That greater efficiency only benefits the LLM SaaS providers as long as the hardware manufacturers, probably especially the VRAM manufacturers, remain supply constrained, since the high-efficiency users are the ones who can pay top dollar. But the hardware guys' dream is presumably to get parts into millions of laptops which remain on standby for 19 hours a day, not to bargain with SaaS providers who obsessively optimise their memory consumption.
I'm not sure if this prediction will hold true.<p>We're not seeing the progress in those "frontier models" that we have previously seen. There's certainly still gas left in tank tank, but we're way into the diminishing returns by now.<p>Cloud inference still beats hardware investments by orders of magnitude of course, but that's only if your data doesn't really matter to you.
We are certainly not in the diminishing returns phase for LLM progress. No sign of that yet.
I’ll grant that for specialized applications like coding agents and mathematics, but even there I suspect that most the real gains are actually taking place in the harness.<p>But I suspect returns may have already diminished into negative territory for at least some other use cases. One of my least favorite job responsibilities in this brave new era is figuring out how to avoid performance and behavior regressions when an older model were using for some application reaches end of life. It’s getting uncommon for me to look at our benchmark results and say, “Oh, good, it does better on one of the newer models!”
>suspect that most the real gains are actually taking place in the harness.<p>Part of the reason harnesses work well is you can run a lot of agents in parallel. That doesn't slow down demand.
That is true, but the eventual realization that more machines doing more coin flips in parallel does not mean "more work gets done" might.<p>LLMs are amazing tech, but they're terrible without oversight. More agents faster just makes reality collapse on them quicker.<p>But yeah, you're right, temporarily, this will still push demand. But the topic was about "diminishing returns" as in "tech getting better". Not as in "customer spending".
But high demand for LLM time isn't sufficient to keep customers at the frontier LLM SaaS providers. That demand can be satisfied locally or at non-frontier outlets, absent hardware shortages at least. The Tier 1 providers (and the would-be Tier 1s) presumably need to open up a much bigger lead in model quality, one that doesn't simply get distilled away this time, and/or continue to be protected by ongoing (or worsening!) hardware shortages. (And that's overlooking the revenue shortfalls which OpenAI and Anthropic seem to be facing already.)
I had actually been thinking more about all the non-LLM functionality that go into the harnesses. I'm not going to name names and I haven't done any rigorous testing, but my general impression is that choice of harness matters more than choice of model. In terms of basic task completion success specifically, not code aesthetics.
One thing that I really want to know - the better models from today vs a year ago - what has changed. They have already pre-trained on all available public data. Scooping up the last percentage of archaic texts which were never digitized is not going to move the needle.<p>Is it just that the providers are generating tons of synthetic datasets on coding tasks so that the models get more exposure to the right thing to do? Every time someone points out an LLM stupidity they add some training data to patch over the weakness (trivial to generate "there are two 'l's in llama")?
Google scholar has a flood of papers showing LLM diminishing returns on pretty much every facet<p><a href="https://scholar.google.com/scholar?hl=en&as_sdt=0%2C23&q=llm+diminishing+returns+&btnG=" rel="nofollow">https://scholar.google.com/scholar?hl=en&as_sdt=0%2C23&q=llm...</a>
Well I mean if I wanted to be extra pedantic, I would argue that we've been in that phase since LLMs were first introduced.<p>Before that, we had 0.
After that, we had more than 1.<p>A leap as far as that is hard to recreate.<p>But that wasn't my point. That's just trolling.<p>The actual point is that LLMs aren't gaining new capabilities anymore. They just get more reliable at the ones they already have; turning what was a coin flip to some higher probability.<p>That's (intuitively speaking, not strictly mathematically speaking) kinda the mathematical definition of diminishing returns.
It’s a constant tension in computing that has been around since mainframes and clients… Neither is going to disappear. My general feeling is normal people care more about how thin and light something is than their privacy, so if data center powered LLMs will have a strong future.
Hmm I'm not 100% sure about that, given that edge is very viable, and the geopolitical climate has changed quite significantly.<p>I agree that datacenters are not going to go away, but I have doubts that the buildup that has happened is really going to pay off for most operators.
they really dont want to hear this bro lol
I can see that by those reddit-style vote swings, but who are "they", exactly?<p>Who is so emotionally invested into random comment sections being purely positive about their pet.. uuuuuuuh.. tech?<p>Very weird.
You don’t need frontier-level performance for every task. That’s why companies hire both junior and senior developers. I suspect a decent percentage of people using Fable would probably be fine with Opus.<p>Also, the frontier can’t keep advancing at this rate forever. Eventually the low-hanging fruit will all be gone and advances will slow down.
It's going to happen very soon, which is why these frontier labs are scrambling to shut down open source language models. There's an existential risk threatening their obscene returns.
It is for that reason they are being archived and torrented as a very large middle finger.
I feel like cost competitiveness of local has been going down, if anything, not up. API providers can use hardware more and have scale efficiencies. Do you see any reason this will reverse?
> It's going to happen very soon<p>Why? You can't just assert it. There are very good reasons to think it <i>won't</i> happen soon, and you've given no reasons to think it will happen soon.
Because everything is converging on a backlog of huge efficiency gains established in research, waiting to be combined. Looped transformers, a whole host of diffusion techniques and new quantization techniques, maturation of ternary distillation and new ways to separate logic from stuff that can be looked up. It would surprise me if most frontier models were actually even that big at that point in terms of active params. I highly doubt it.
You asserted that it was never going to happen first
Or even more amusing in time for the bubble to burst, I hope the big RAM makers end up holding the whole bag for that. Greedy bastards.
Nearly the entire reason we're in this mess is because RAM/SSD/HDD makers are terrified of AI bubble bursting.<p>Long-term contracts, not building new fabs - they're in full on hedging mode right now.
Seemingly the AI bulls are either not sufficiently optimistic, or not sufficiently wealthy(?!), to build new fabs as joint ventures with the manufacturers in which they agree to assume most of the downside risk?
They signed long term contracts for the hardware delivery over years. AI investment is approaching 1 trillion per year and expected to grow to well over 1 trillion per year. [1]<p>Even over their many years, projects like the Manhattan Project, the Apollo Program, or the U.S. Interstate Highway System never added up to that. [2]<p>Is your argument that there is insufficient AI spending at present as they aren’t also taking on building their own fabs?<p>[1] <a href="https://www.pwc.com/gx/en/news-room/press-releases/2026/global-investment-in-ai-infrastructure.html" rel="nofollow">https://www.pwc.com/gx/en/news-room/press-releases/2026/glob...</a><p>[2] <a href="https://www.aljazeera.com/news/2026/2/19/visualising-ai-spending-how-does-it-compare-with-historys-mega-projects" rel="nofollow">https://www.aljazeera.com/news/2026/2/19/visualising-ai-spen...</a>
If the long term contact are with any AI companies, then good luck getting anything back in bankrupcy proceedings!
If the AI bubble bursts, the DRAM manufacturers switch back to making conventional DRAM and pent-up business and consumer demand results in a surge of purchases, cushioning the fall considerably.
Looking forward to see them burn.
Since you’re not greedy when there is a memory glut I’m sure you’ll be willing to pay extra to make it fair.
Not at all, I’m going to relax with a cooling drink and watch the consequences of this unbelievably destructive and wasteful venture implode. I also adore the idea that it's on the customer to be "fair" to a company that dumped us a group in favor of chasing B2B money.<p>Their choice, their consequences.
I think they're talking about the past: memory has often been boom/bust. Memory manufacturers lose money during gluts, and did so long before AI. People benefited from cheap prices at that time.
Internet is a fad, it will implode at any moment.
In fact as a business opportunity it did just that because the massive investment in the mid-late 1990's was without anything like a connection to profitability or sustainable demand. It took years after that crash for the concept of e-commerce to begin both a recovery and evolution into what we see today.<p>I expect something similar for AI. It's useful tech... just not very profitable tech, and certainly not to the tune of trillions of dollars worth of public demand. The promises of superintelligence, replacing everyone, and the rest of the hype will die with the companies who made the promises, but the tech will survive and thrive.
>The promises of superintelligence, replacing everyone, and the rest of the hype will die with the companies who made the promises<p>The hype will die but there are many reasons why super intelligence and robots are a separate entity from said hype.<p>Look, back a few decades and tell someone our gdp would be in the trillions and it's likely they'd have a hard time believing you. A huge portion of our products that we use day to day would be complete science fiction to them.<p>All we're negotiating at this point is the timescale it will take.
Samsung is already building another fab (which was originally suspended due to <i>low</i> memory prices years ago). Hopefully when it goes live in a few years it's not all dedicated to HBM for AI.
Not even close.
Not until the industry gets severe indigestion, which doesn't seem that far off.
Good to see Samsung stepping up; maybe GPU prices will finally cool down a bit with more HBM supply hitting the market.
A shortage is almost always followed by a glut. Memory of all kinds is going to get very cheap due to supply and demand and substitution effects.
You just know that the AI bubble is going to break and they're going to be stuck with tons of HBM4s.<p>The big problem is that this will not depress the consumer DRAM market and mean we can all afford DRAM again
Good, maybe this means cloud GPU prices will <i>eventually</i> normalize. Still feeling the pinch from the AI gold rush.
This insane nonsence needs to stop.
To me, the most idiotic thing is that the world currently cannot build power plants fast enough to even use all the HBM that is being produced. That is, a large portion of HBM and cards are sitting in storage, waiting for the data centers and the power plants to run them. The FOMO in the AI world isn't just in the stock market
This is crazy!
Now make some DDR5.
[dead]
[dead]
[flagged]
The beginning of the bubble pop?
About 2 years ago I bought some SDRAM or something like that, DDR4 or DDR5, don't recall offhand. A few months ago I looked at the price today, and it was over
3x as high. That's just insane. Governments need to do something against this abuse system that AI amplified here.
> Governments need to do something against this abuse system that AI amplified here.<p>Which governments, and what exactly would you want these governments to do? The demand is global. There are no levers a single government can pull to meaningfully influence the global demand without fully committing to an protectionist economic policy, in which case the U.S. doesn't have the facilities to magically pop up world class fabs overnight, and South Korea and Taiwan don't have the market demand that the U.S. generates to justify their investments in making these chips and China lacks the IP to be able to build anything comparable to Nvidia's silicon at the moment.<p>No one has all the cards and no one controls all the levers.
Governments have sued DRAM companies for price fixing and forming cartels before: <a href="https://en.wikipedia.org/wiki/DRAM_industry_price_fixing" rel="nofollow">https://en.wikipedia.org/wiki/DRAM_industry_price_fixing</a>. They can absolutely step in and if there are evidence of price fixing or anti-competitive behaviours, can impose fines. The choice of whether to do that though is political.
And on that page govts have tried and failed at it also. Thinking the current prices are the result of fixing and not insane demand is surely going to lose in court. It’s trivial to show massive demand, and thus production line changes, as the driving forces here.<p>If you look over all such lawsuits, they extremely rarely succeed.
Several of these companies almost went under before the AI boom. There isn’t a cartel.
Is there any allegations of price fixing in this case? The demand just skyrocketed. Other than legislation like the defense act in the US I don’t know of any governments that have jurisdiction over these ram makers that mandate companies produce more of a product.<p>Especially not for consumer goods.
The same major memory makers got prosecuted for cartel behavior by the DoJ a little over 20 years ago[0].<p>One might say "well what specific law have they broken" and I don't know and I don't know if they have, but at the very least the power of the State _could_ be used to force behavioral changes in some ways or another.<p>[0]: <a href="https://en.wikipedia.org/wiki/DRAM_industry_price_fixing" rel="nofollow">https://en.wikipedia.org/wiki/DRAM_industry_price_fixing</a>
The same governments that dumped bilions into to these companies and gave them insane tax breaks.
<i>No one has all the cards and no one controls all the levers.</i><p>I'm as fanatical a free-market fundamentalist as you'll ever meet, but if someone were to argue that government has a role in preventing bullshit like OpenAI's unilateral 40% attack on the entire DRAM market, backed by nothing but funny money, I would have a hard time coming up with defensible counterarguments.
I don’t really know much about how this stuff works, but I have a feeling we wouldn’t even have to do anything punitive. We might just need to find and close whatever weird loophole allows the folks who are participating in this gold rush to feel confident enough that they’ve externalized their risks to be willing to engage in speculative data center buildout projects on such a grand scale in the first place.
Apple used to do the same thing with TSMC’s newer process nodes, and nobody really complained because people were getting superior products. I happen to like US antitrust enforcement that considers consumer benefit primarily.
Your attempts to insult what they are doing doesn't show they did anything wrong, but it does impeach the validity of your own argument.
How much memory does it take for the consumer market not to be totally screwed?<p>Can that share of production be allocated to consumers, and the AI fights over the rest?
It is funny how regular people are trapped into legalism while companies are seeing things as a two sided fight.<p>You can’t demand anything with this mentality.<p>“The current situation is messing up the economy for a very large portion of us” is universally a good enough reason
What abuse? Responding to supply and demand fluctuations isn't abuse, it's the proper operation of the market.
Would if one industry gobbles up an important resource to the point that other consumers lose practical access to it, shouldn't there be limits?
No. The other industries can bid for access to the resource, and under most circumstances, whether they are willing to bid higher tracks importance. Sometimes there are cases where something has significant positive externalities and so people are not willing to bid high enough, and then we can talk about some kind of government intervention (typically a subsidy that attempts to track the value of the externalities) but I don't see that applying here.
If others are willing to pay more, that a signal that more value is created if the resource goes to them rather than to the more price sensitive customers.
This logic falls apart completely when gamblers allocate one trillion to bidding up the price, denying access to the resource to people who are responsibly spending their own money. The ideal world of the imaginary perfect free market never takes into consideration the messiness of the real world, like the fact that people will spend extreme sums of money irrationally. It can take years for this effect to correct, damaging the market severely in the meantime.<p>Alternatively, the money invested can be rational because it prices out competitors and establishes a monopoly, after which point the monopolist earns their absurd investment back with complete control of the market. This is <i>also</i> bad.
You can reach any conclusion you want if you assume others are behaving irrationally. But why should I assume they are wrong instead of you being wrong?
This has zero to do with the company that is making the product and all to do with the company buying them.<p>The problem you have here is hundreds of different companies are doing the "gambling" in a non collusionary manner so it's going to take decades in court to prove it. Are you saying governments should do an authoritarian take over of RAM allotment?
Where is the additional supply resulting from the sustained increase in demand?
Some are building new fabs, example:<p><a href="https://news.skhynix.com/en/fab-facility-investment-2026/" rel="nofollow">https://news.skhynix.com/en/fab-facility-investment-2026/</a><p>It's not an overnight thing obviously.
Check the article we're commenting under
3x? I bought 128GB DDR5 SODIMMs for 300 eur. Now it's 4000 eur.
It's not just normal consumer memory. Everything with a chip in it is getting more expensive because of the AI circlejerk. When Bitcoin screwed over consumers, at least normal memory prices weren't affected (though that time a shitcoin based around storing large amounts of data made the rounds, hard drive prices did take a brief hit).<p>Everything is affected by the endless thirst for RAM by AI companies and the accompanying hyperscalers. I'm honestly disappointed governments aren't doing more to keep the price of consumer goods in check.<p>I guess the AI companies telling the world how good they are at accidental cybercrime and fraud is creating enough FUD to let the AI industry run wild.