Run an incredible 400B parameters on a handheld device.<p>0.6 t/s, wait 30 seconds to see what these billions of calculations get us:<p>"That is a profound observation, and you are absolutely right ..."
Better than waiting 7.5 million years to have a tell you the answer is 42.
I don't think we are ever going to win this. The general population loves being glazed way too much.
> The general population loves being glazed way too much.<p>This is 100% correct!
That's an astute point, and you're right to point it out.
I mean size says nothing, you could do it on a Pi Zero with sufficient storage attached.<p>So this post is like saying that yes an iPhone is Turing complete. Or at least not locked down so far that you're unable to do it.
I thought you were being sarcastic until I watched the video and saw those words slowly appear.<p>Emphasis on slowly.
> SSD streaming to GPU<p>Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]?<p>1: <a href="https://arxiv.org/abs/2312.11514" rel="nofollow">https://arxiv.org/abs/2312.11514</a>
Yes. I collected some details here: <a href="https://simonwillison.net/2026/Mar/18/llm-in-a-flash/" rel="nofollow">https://simonwillison.net/2026/Mar/18/llm-in-a-flash/</a>
This is not entirely dissimilar to what Cerebus does with their weights streaming.
A similar approach was recently featured here: <a href="https://news.ycombinator.com/item?id=47476422">https://news.ycombinator.com/item?id=47476422</a> Though iPhone Pro has very limited RAM (12GB total) which you still need for the active part of the model. (Unless you want to use Intel Optane wearout-resistant storage, but that was power hungry and thus unsuitable to a mobile device.)
> Though iPhone Pro has very limited RAM (12GB total) which you still need for the active part of the model.<p>This is why mixture of experts (MoE) models are favored for these demos: Only a portion of the weights are active for each token.
Yeah, this new post is a continuation of that work.
It’s 400B but it’s mixture of experts so how many are active at any time?
Looks like it's Qwen3.5-397B-A17B so 17B active. <a href="https://github.com/Anemll/flash-moe/tree/iOS-App" rel="nofollow">https://github.com/Anemll/flash-moe/tree/iOS-App</a>
Aren't most companies doing MoE at this point?
This is awesome! How far away are we from a model of this capability level running at 100 t/s? It's unclear to me if we'll see it from miniaturization first or from hardware gains
On smartphones? It’s not worth it to run a model this size on a device like this. A smaller fine-tuned model for specific use cases is not only faster, but possibly more accurate when tuned to specific use cases. All those gigs of unnecessary knowledge are useless to perform tasks usually done on smartphones.
Only way to have hardware reach this sort of efficiency is to embed the model in hardware.<p>This exists[0], but the chip in question is physically large and won't fit on a phone.<p>[0] <a href="https://www.anuragk.com/blog/posts/Taalas.html" rel="nofollow">https://www.anuragk.com/blog/posts/Taalas.html</a>
A year ago this would have been considered impossible. The hardware is moving faster than anyone's software assumptions.
This isn't a hardware feat, this is a software triumph.<p>They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).
It's both.<p>We haven't had phones running laptop-grade CPUs/GPUs for that long, and that is a very real hardware feat. Likewise, nobody would've said running a 400b LLM on a low-end laptop was feasible, and that is very much a software triumph.
The iPhone 17 Pro launched 8 months ago with 50% more RAM and about double the inference performance of the previous iPhone Pro (also 10x prompt processing speed).
It wasn't considered impossible. There are examples of large MoE LLMs running on small hardware all over the internet, like giant models on Raspberry Pi 5.<p>It's just so slow that nobody pursued it seriously. It's fun to see these tricks implemented, but even on this 2025 top spec iPhone Pro the output is 100X slower than output from hosted services.
The software has real software engineers working on it instead of researchers.<p>Remember when people were arguing about whether to use mmap? What a ridiculous argument.<p>At some point someone will figure out how to tile the weights and the memory requirements will drop again.
<a href="https://xcancel.com/anemll/status/2035901335984611412" rel="nofollow">https://xcancel.com/anemll/status/2035901335984611412</a>
[dead]
[flagged]
This has nothing to do with Apple, and everything to do with MoE and that everyone forgot you can re-read the necessary bits of the model from disk for each token.<p>This is extremely inefficient though. For efficiency you need to batch many requests (like 32+, probably more like 128+), and when you do that with MoE you lose the advantage of only having to read a subset of the model during a single forward pass, so the trick does not work.<p>But this did remind me that with dense models you might be able to use disk to achieve high throughput at high latency on GPUs that don't have a lot of VRAM.
Apple might just win the AI race without even running in it. It's all about the distribution.
Because someone managed to run LLM on an iPhone at unusable speed Apple won AI race? Yeah, sure.
Apple is already one of the winners of the AI race. It’s making much more profit (ie it ain’t losing money) on AI off of ChatGPT, Claude, Grok (you would be surprised at how many incels pay to make AI generated porn videos) subscriptions through the App Store.<p>It’s only paying Google $1 billion a year for access to Gemini for Siri
It's crazy to see a 400B model running on an iPhone. But moving forward, as the information density and architectural efficiency of smaller models continue to increase, getting high-quality, real-time inference on mobile is going to become trivial.