For anyone wondering “how slow is this?”<p>IIUC, Kimi K3 on RTX 6000 Ada (48GB) takes 292 s/token<p><a href="https://github.com/lyogavin/airllm/releases/tag/v3.1.0" rel="nofollow">https://github.com/lyogavin/airllm/releases/tag/v3.1.0</a>
I do think these “run a bigger model than will fit in VRAM” projects are necessary steps, but are they functionally useful or helpful to anyone currently? For example, is anyone out there running a big Qwen for coding on a 16-32GB machine with these techniques?
I don't need any of these to help with coding. skill issue.
> but are they functionally useful or helpful to anyone currently?<p>Yes and no, depend on your expectations. Some/many like to run local LLMs just for the sake of it, so anything will do.<p>MoE are useful on PC systems, at the condition of having high enough memory bandwidth (and large amounts of RAM) - that is, Threadripper/Pro.<p>The advantage of MoE is that only a subset of the model's experts is used for each token, so not all weights need to be present in VRAM at once. The remaining weights can reside in system RAM, although moving and accessing them still carries a substantial performance cost (and that's why high memory bandwidth is needed).
Does MoE help with multimodality? Can it in general enable reasoning in imagery (technical drawings, diagrams, schematics) rather than text-based?
MoE has nothing to do with multimodality.<p>MoE is a concept proposed in 1991, before the deep learning era (which is before what I call the transformers era). You can think of it like sharing.<p>Contrary to popular belief; 'experts' in MoE LLMs do not specialize. There's no expert trained to be good at maths, or python, or writing, or whatever. It's an inference optimization.<p>As for reasoning in non-text modalities, you might find this paper interesting :) <a href="https://huggingface.co/papers/2502.05171" rel="nofollow">https://huggingface.co/papers/2502.05171</a>
Wait I thought the router ends up specializing the experts?<p>Like there is no explicit goal aside from each 'expert' getting roughly equal weight?<p>And it happens that when you train the router you do end up passing certain classes of problem to each expert - just as a training result nothing as clean as a python expert. But math vs creative writing will tend to rely on different experts over the majority of the inference?<p>I do not know what I am talking about, this is my limited understanding...
> You can think of it like sharing.<p>was this meant to read "sharding"?
Are people getting decent tokens/second throughput? Some of these demos crawl at 1 tok/s or worse, which limits their utility.
yes, I average 80-120 tok/s on my RTX 3080 with gemma 4 and faster with Qwen 3.5. The main use-case here is just code-monkey agents. I'm not looking for architectural guidance, but an agent to take a spec and complete it.
in theory, QWEN3.6-35B-A3B should run as fast as a 3B model; and in practice, it can be just as dumb.
If I could justify wear and tear and electricity, I was willing to do something like this for batch processing. The batches would be a bunch of prompts whose outputs I'd look at the next day. Maybe common operations, like QA or refactoring, on whatever software I wrote.<p>If so, I could use a larger model than I have real-time hardware for. The largest, well-trained models can often get the output mostly right in one try. I also would be using AI's as a supplement to, not replacement for, my own brain. So, issues with the outputs wouldn't be a problem because I'm just keeping what's helpful.<p>If I still need to re-generate it all, it might still save money over time by avoiding cloud costs. Also, hardware that's already paid for is a sunk cost that doesn't inflate over time. Glitches in loading or destroying VM's might blow up into a big bill.
At that point, how does this compare with simply running the model on the CPU?
It matches my coding speed...its ok.
Ahaha thank you, I naively assumed the unlabeled graph in the readme was tps, not spt!
I wonder what this measures in J/token.
How many is that in tokens per Scaramucci?
that's 0.003 tokens/second. To get an hour's work done that's normally 30 tokens/second (108k output tokens in an hour) will take 416 days at this rate. And if you're using 100 watts, during that time you will spend $124.61 in electricity, as well as not being able to use your device for something else, plus the noise and heat from your device.<p>For $124, on Moonshot's official Kimi K3 API rates ($0.30 per 1M cached input, $3 per 1M fresh input, $15 per 1M fresh output), you can purchase 42 million fresh-input tokens, or 8.3 million generated output tokens, in whatever mix you want.<p>So what you get is 80x more expensive and you wait 416 days to get it.
Wow, you could do a <i>lot</i> at 292 tokens a sec—oh.<p>I have all praise for those taking this on and in my idiom would call it *the lord's work."<p>The image I reliably summon to mind is that compilation video showing the progress of Boston Dynamics bots. The curve between technically functional, to comically slow, to too slow for "real" work, on to, OMFG, may prove a (rough) curve.<p>It's work like this that moves things forward.
Hah I was looking for it and couldn't work out how many years/token. 292s is pretty good.