I'm still sad that we haven't seen a new Taalas style chip a la <a href="https://chatjimmy.ai/" rel="nofollow">https://chatjimmy.ai/</a>. Smaller models are good enough now to make that insane burst of tokens so useful.
If you care about speed Cerebras gpt-oss-120b is 1400tk/s and "just as smart" in ranking.<p>I've used it on a few for fun projects and its decent but the speed is crazy to watch.
I felt happy I could run it at 100tk/s on my new Mac Studio :')
Also kimi 2.6 at 1000tps (as of may), though when we reached out they had a >12 month waitlist and minimum 7-8 figure annual token spend.<p>[0] <a href="https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise" rel="nofollow">https://www.cerebras.ai/blog/cerebras-kimi-k2-Enterprise</a>
7-8 figures annual spend will buy a hell of a lot of capable local inference hardware you can own, though it won't be at the absurd token/s rate, you'll be able to run almost <i>anything</i> on it... And it'll still have a good residual resale value after 4 years the way things are going now.
yeah. K2.6 can run on insane speeds. So sad that they don't have K3 yet.<p>But it can apparently also run 5.6 Sol
It was better when they had gemma at 1k. Inco does DS flash at about 600. A few places will do K3 and GLM in the hundreds.<p>Such a tiny model at that t/s is less impressive than it would have been four months ago.
if you want to feel the speed without the burn<p><a href="https://kamilstanuch.github.io/LLM-token-generation-simulator/" rel="nofollow">https://kamilstanuch.github.io/LLM-token-generation-simulato...</a>
Please do not try to use gpt-oss-120b over Cerebras. It is broken, screws up tool calls most of the time, forgets to end thinking blocks and has all sorts of other issues. The speed is amazing but it is absolutely not worth it, especially at that quite incredible cost. Think: $5–10/minute levels of cost with a single agent, because Cerebras also offers no cache pricing for input tokens at all.
Which is wild because it does, in fact, do caching
Yea i had some pretty meh results using gpt-oss-120b it in my evals where it should have benefited speed alot but it really under performed what i was expecting.
Or better: Qwen 2.8 27b
Chat Jimmy clocks at 17K tokens per sec burning LLM into the Chip - <a href="https://chatjimmy.ai/" rel="nofollow">https://chatjimmy.ai/</a> - Source: <a href="https://theashishmaurya.medium.com/taalas-the-startup-that-prints-ai-models-directly-onto-silicon-33b181690575" rel="nofollow">https://theashishmaurya.medium.com/taalas-the-startup-that-p...</a>
At some point the bottleneck becomes tool calling.. and as such, it's preferably if the model is co-hosted (in the same datacenter, at least) with your code repository and all other reference/context it needs (full documentation for most ecosystems, maybe even a copy of common crawl to minimize web fetch usage, etc)
Pricing at $0.25 and $0.75 already puts its cost well above reasonably reputable inference providers for deepseek v4 flash or qwen 3.8-flash-next or similar class of open weight LLMs that fit in under 170GB of RAM, so I don't see the point. I think this is probably also stupider than laguna s 2.1 which can also be very cheap to serve.
I have tried using Mercury 2.5 for a lot of my tasks.. but this model just isn't there. It seems to be on par with any 14B model at max. Even GPT-OSS-20B performs way better than this in my own attempts to use it.<p>I really really wanted to use this because it offers incredible speeds and pricing combinations. But nop.. I still am not using it.. not even for basic tasks.
Cerebras is fast…?
I honestly think the diffusion LLM approach is a dead end<p>It's telling that frontier labs like Google toyed around with it but didn't invest further even for their most speed and cost sensitive small models<p>Still unclear for what, if any use cases this is pareto frontier
You can't think that a small startup versus Anthropic's training setup is anywhere near the same scale to make apples to apples comparisons.<p>Not sure how the Chinese labs pull it off though using autoregressive models. The secret sauce is probably going to be in the training data.<p>The main reason Google hasn't switched over to DiffusionGemma is because serving at larger batch sizes loses the speed gains you get from diffusion, and most of the primary use case is serving many users at once off a single device with a large batch size.<p>If you were to move to on-device low latency... like say in a robot or something, then the story might be different...
From what I’ve heard, the issue is more that it’s harder to efficiently share the hardware across diffusion requests, so it’s more expensive to serve.<p>It sounds like there might be opportunities for local models (not open weight, but actually locally run) to use diffusion for faster responses on weaker hardware that doesn’t need to be shared.<p>But yea, it’s still a red-ish flag that big labs haven’t invested much in it. I could see Google/Apple getting value of this sort of local model, but maybe there’s enough research behind traditional models that it’s not worth the distraction at this point in time.
DiffusionGemma was released alongside the other gemma-4 models just a few months ago, so clearly google hasn't abandoned the idea.<p>K2-Horizon-7B has a diffusion and non-diffusion variant, and they claim the same level of intelligence from both models.
Personally, I think it's more that text diffusion is not the ideal driver of an agentic work loop than that text diffusion is a total dead end. I am still hoping to see how it does on authoring and editing with further scaling and optimization. I think the push for AGI has put a bit too much focus on the idea of <i>one</i> general model doing everything.
this feels like "we got the same benches as gpt-oss-120b but are also potentially slower while saying it is great"
I used this a few days ago and thought something must be wrong with how fast it was responding. "Mercury 2.5 is below average in intelligence, but well priced when comparing to other models of similar price." this is so funny. So when you have a stupid model that is fast - what do you use it for?
it's stupid fast!
> Mercury 2.5 is below average in intelligence, but well priced when comparing to other models of similar price.<p>Well priced when compared to other models of similar price, eh?<p>Are we allowed to call this slop, even if the output is not directly from an LLM?
The speed means absolutely nothing when it is finishing almost dead last when compared to the frontier AI companies.
Ah, the old "good, fast, or cheap; pick two" proves true once again.
It means something, because it an iterative workflow. If you're willing to burn tokens, it's possible for weaker models to implement tasks by incrementally improving drafts.
Some of us want fast food
Not if your use case needs speed. For one of my products I can't use an LLM that has a p99 of >700ms for TTFT.