This is not a general purpose chip but specialized for high speed, low latency inference with small context. But it is potentially a lot cheaper than Nvidia for those purposes.<p>Tech summary:<p><pre><code> - 15k tok/sec on 8B dense 3bit quant (llama 3.1)
- limited KV cache
- 880mm^2 die, TSMC 6nm, 53B transistors
- presumably 200W per chip
- 20x cheaper to produce
- 10x less energy per token for inference
- max context size: flexible
- mid-sized thinking model upcoming this spring on same hardware
- next hardware supposed to be FP4
- a frontier LLM planned within twelve months
</code></pre>
This is all from their website, I am not affiliated. The founders have 25 years of career across AMD, Nvidia and others, $200M VC so far.<p>Certainly interesting for very low latency applications which need < 10k tokens context. If they deliver in spring, they will likely be flooded with VC money.<p>Not exactly a competitor for Nvidia but probably for 5-10% of the market.<p>Back of napkin, the cost for 1mm^2 of 6nm wafer is ~$0.20. So 1B parameters need about $20 of die. The larger the die size, the lower the yield. Supposedly the inference speed remains almost the same with larger models.<p>Interview with the founders:
<a href="https://www.nextplatform.com/2026/02/19/taalas-etches-ai-models-onto-transistors-to-rocket-boost-inference/" rel="nofollow">https://www.nextplatform.com/2026/02/19/taalas-etches-ai-mod...</a>
This math is useful. Lots of folks scoffing in the comments below. I have a couple reactions, after chatting with it:<p>1) 16k tokens / second is really stunningly fast. There’s an old saying about any factor of 10 being a new science / new product category, etc. This is a new product category in my mind, or it could be. It would be incredibly useful for voice agent applications, realtime loops, realtime video generation, .. etc.<p>2) <a href="https://nvidia.github.io/TensorRT-LLM/blogs/H200launch.html" rel="nofollow">https://nvidia.github.io/TensorRT-LLM/blogs/H200launch.html</a> Has H200 doing 12k tokens/second on llama 2 12b fb8. Knowing these architectures that’s likely a 100+ ish batched run, meaning time to first token is almost certainly slower than taalas. Probably much slower, since Taalas is like milliseconds.<p>3) Jensen has these pareto curve graphs — for a certain amount of energy and a certain chip architecture, choose your point on the curve to trade off throughput vs latency. My quick math is that these <i>probably do not</i> shift the curve. The 6nm process vs 4nm process is likely 30-40% bigger, draws that much more power, etc; if we look at the numbers they give and extrapolate to an fp8 model (slower), smaller geometry (30% faster and lower power) and compare 16k tokens/second for taalas to 12k tokens/s for an h200, these chips are in the same ballpark curve.<p>However, I don’t think the H200 can reach into this part of the curve, and that does make these somewhat interesting. In fact even if you had a full datacenter of H200s already running your model, you’d probably buy a bunch of these to do speculative decoding - it’s an amazing use case for them; speculative decoding relies on smaller distillations or quants to get the first N tokens sorted, only when the big model and small model diverge do you infer on the big model.<p>Upshot - I think these will sell, even on 6nm process, and the first thing I’d sell them to do is speculative decoding for bread and butter frontier models. The thing that I’m really very skeptical of is the 2 month turnaround. To get leading edge geometry turned around on arbitrary 2 month schedules is .. ambitious. Hopeful. We could use other words as well.<p>I hope these guys make it! I bet the v3 of these chips will be serving some bread and butter API requests, which will be awesome.
> any factor of 10 being a new science / new product category,<p>I often remind people two orders of quantitative change is a qualitative change.<p>> The thing that I’m really very skeptical of is the 2 month turnaround. To get leading edge geometry turned around on arbitrary 2 month schedules is .. ambitious. Hopeful. We could use other words as well.<p>The real product they have is automation. They figured out a way to compile a large model into a circuit. That's, in itself, pretty impressive. If they can do this, they can also compile models to an HDL and deploy them to large FPGA simulators for quick validation. If we see models maturing at a "good enough" state, even a longer turnaround between model release and silicon makes sense.<p>While I also see lots of these systems running standalone, I think they'll really shine combined with more flexible inference engines, running the unchanging parts of the model while the coupled inference engine deals with whatever is too new to have been baked into silicon.<p>I'm concerned with the environmental impact. Chip manufacture is not very clean and these chips will need to be swapped out and replaced at a cadence higher than we currently do with GPUs.
Having dabbled in VLSI in the early-2010s, half the battle is getting a manufacturing slot with TSMC. It’s a dark art with secret handshakes. This demonstrator chip is an enormous accomplishment.
There might be a foodchain of lower order uses when they become "obsolete".
I think there will be a lot of space for sensorial models in robotics, as the laws of physics don't change much, and a light switch or automobile controls have remained stable and consistent over the last decades.
I think the next major innovation is going to be intelligent model routing. I've been exploring OpenClaw and OpenRouter, and there is a real lack of options to select the best model for the job and execute. The providers are trying to do that with their own models, but none of them offer everything to everyone at all times. I see a future with increasingly niche models being offered for all kinds of novel use cases. We need a way to fluidly apply the right model for the job.
Agree that routing is becoming the critical layer here. Vllm iris is really promising for this <a href="https://blog.vllm.ai/2026/01/05/vllm-sr-iris.html" rel="nofollow">https://blog.vllm.ai/2026/01/05/vllm-sr-iris.html</a><p>There's already some good work on router benchmarking which is pretty interesting
At 16k tokens/s why bother routing? We're talking about multiple orders of magnitude faster and cheaper execution.<p>Abundance supports different strategies. One approach: Set a deadline for a response, send the turn to every AI that could possibly answer, and when the deadline arrives, cancel any request that hasn't yet completed. You know a priori which models have the highest quality in aggregate. Pick that one.
I came across this yesterday. Haven't tried it, but it looks interesting:<p><a href="https://agent-relay.com/" rel="nofollow">https://agent-relay.com/</a>
[dead]
For speculative decoding, wouldn’t this be of limited use for frontier models that don’t have the same tokenizer as Llama 3.1? Or would it be so good that retokenization/bridging would be worth it?
My understanding as well is that speculative decoding only works with a smaller quant of the same model. You're using the faster sampling of the smaller models representation of the larger models weights in order to attempt to accurately predict its token output. This wouldn't work cross-model as the token probabilities are completely different.
This is not correct.<p>Families of model sizes work great for speculative decoding. Use the 1B with the 32B or whatever.<p>It's a balance as you want it to be guessing correctly as much as possible but also be as fast as possible. Validation takes time and every guess needs to be validated etc<p>The model you're using to speculate could be anything, but if it's not guessing what the main model would predict, it's useless.
Smaller quant or smaller model?<p>Afaik it can work with anything, but sharing vocab solves a lot of headaches and the better token probs match, the more efficient it gets.<p>Which is why it is usually done with same family models and most often NOT just different quantizations of the same model.
I think they’d commission a quant directly. Benefits go down a lot when you leave model families.
The guts of a LLM isn't something I'm well versed in, but<p>> to get the first N tokens sorted, only when the big model and small model diverge do you infer on the big model<p>suggests there is something I'm unaware of. If you compare the small and big model, don't you have to wait for the big model anyway and then what's the point? I assume I'm missing some detail here, but what?
Speculative decoding takes advantage of the fact that it's faster to validate that a big model would have produced a particular sequence of tokens than to generate that sequence of tokens from scratch, because validation can take more advantage of parallel processing. So the process is generate with small model -> validate with big model -> then generate with big model only if validation fails<p>More info:<p>* <a href="https://research.google/blog/looking-back-at-speculative-decoding/" rel="nofollow">https://research.google/blog/looking-back-at-speculative-dec...</a><p>* <a href="https://pytorch.org/blog/hitchhikers-guide-speculative-decoding/" rel="nofollow">https://pytorch.org/blog/hitchhikers-guide-speculative-decod...</a>
Verification is faster than generation, one forward pass for verification of multiple tokens vs a pass for every new token in generation
I don't understand how it would work either, but it may be something similar to this: <a href="https://developers.openai.com/api/docs/guides/predicted-outputs" rel="nofollow">https://developers.openai.com/api/docs/guides/predicted-outp...</a>
They are referring to a thing called "speculative decoding" I think.
When you predict with the small model, the big model can verify as more of a batch and be more similar in speed to processing input tokens, if the predictions are good and it doesn't have to be redone.
Think about this for solving questions in math where you need to explore a search space. You can run 100 of these for the same cost and time of doing one api call to open ai.
In 20$ a die, they could sell Gameboy style cartridges for different models.
That would be very cool, get an upgraded model every couple of months. Maybe PCIe form factor.
Make them shaped like floppy disks to confuse the younger generations.
Were we go towards really smart roboters. It is interesting what kind of diferent model chips they can produce.
Don’t forget that the 8B model requires 10 of said chips to run.<p>And it’s a 3bit quant. So 3GB ram requirement.<p>If they run 8B using native 16bit quant, it will use 60 H100 sized chips.
> 880mm^2 die<p>That's a lot of surface, isn't it? As big an M1 Ultra (2x M1 Max at 432mm² on TSMC N5P), a bit bigger than an A100 (820mm² on TSMC N7) or H100 (814mm² on TSMC N5).<p>> The larger the die size, the lower the yield.<p>I wonder if that applies? What's the big deal if a few parameter have a few bit flips?
There’s a bit of a hidden cost here… the longevity of GPU hardware is going to be longer, it’s extended every time there’s an algorithmic improvement. Whereas any efficiency gains in software that are not compatible with this hardware will tend to accelerate their depreciation.
Sounds perfect for use in consumer devices.
Do not overlook traditional irrational investor exuberance, we've got an abundance of that right now. With the right PR manouveurs these guys could be a tulip craze.
Yea its fast af but very quickly loses context/hallucinates from my own tests with large chunks of text
This is insane if true - could be super useful for data extraction tasks. Sounds like we could be talking in the cents per millions of tokens range.
Low-latency inference is a huge waste of power; if you're going to the trouble of making an ASIC, it should be for dog-slow but very high throughput inference. Undervolt the devices as much as possible and use sub-threshold modes, multiple Vt and body biasing extensively to save further power and minimize leakage losses, but also keep working in fine-grained nodes to reduce areas and distances. The sensible goal is to expend the least possible energy per operation, even at increased latency.
Low latency inference is very useful in voice-to-voice applications. You say it is a waste of power but at least their claim is that it is 10x more efficient. We'll see but if it works out it will definitely find its applications.
I think it's really useful for agent to agent communication, as long as context loading doesn't become a bottleneck. Right now there can be noticeable delays under the hood, but at these speeds we'll never have to worry about latency when chain calling hundreds or thousands of agents in a network (I'm presuming this is going to take off in the future). Correct me if I'm wrong though.
Doesn't the blog state that it's now 4bit (the first gen was 3bit + 6bit)?
An on-device reasoning model what that kind of speed and cost would completely change the way people use their computers. It would be closer to star trek than anything else we've ever had. You'd never have to type anything or use a mouse again.
What's happening in the comment section? How come so many cannot understand that his is running Llama 3.1 8B? Why are people judging its accuracy? It's almost a 2 years old 8B param model, why are people expecting to see Opus level response!?<p>The focus here should be on the custom hardware they are producing and its performance, that is whats impressive. Imagine putting GLM-5 on this, that'd be insane.<p>This reminds me a lot of when I tried the Mercury coder model by Inceptionlabs, they are creating something called a dLLM which is like a diffusion based llm. The speed is still impressive when playing aroun with it sometimes. But this, this is something else, it's almost unbelievable. As soon as I hit the enter key, the response appears, it feels instant.<p>I am also curious about Taalas pricing.<p>> Taalas’ silicon Llama achieves 17K tokens/sec per user, nearly 10X faster than the current state of the art, while costing 20X less to build, and consuming 10X less power.<p>Do we have an idea of how much a unit / inference / api will cost?<p>Also, considering how fast people switch models to keep up with the pace. Is there really a potential market for hardware designed for one model only? What will they do when they want to upgrade to a better version? Throw the current hardware and buy another one? Shouldn't there be a more flexible way? Maybe only having to switch the chip on top like how people upgrade CPUs. I don't know, just thinking out loudly.
They don't give cost figures in their blog post but they do here:<p><a href="https://www.nextplatform.com/wp-content/uploads/2026/02/taalas-hci-performance-3.jpg" rel="nofollow">https://www.nextplatform.com/wp-content/uploads/2026/02/taal...</a><p>Probably they don't know what the market will bear and want to do some exploratory pricing, hence the "contact us" API access form. That's fair enough. But they're claiming orders of magnitude cost reduction.<p><i>> Is there really a potential market for hardware designed for one model only?</i><p>I'm sure there is. Models are largely interchangeable especially as the low end. There are lots of use cases where you don't need super smart models but cheapness and fastness can matter a lot.<p>Think about a simple use case: a company has a list of one million customer names but no information about gender or age. They'd like to get a rough understanding of this. Mapping name -> guessed gender, rough guess of age is a simple problem for even dumb LLMs. I just tried it on ChatJimmy and it worked fine. For this kind of exploratory data problem you really benefit from mass parallelism, low cost and low latency.<p><i>> Shouldn't there be a more flexible way?</i><p>The whole point of their design is to sacrifice flexibility for speed, although they claim they support fine tunes via LoRAs. LLMs are already supremely flexible so it probably doesn't matter.
I personally don't buy it, cerebras is way more advanced than this, comparing this tok/s to cerebras is disingenious.
If I could have one of these cards in my own computer do you think it would be possible to replace claude code?<p>1. Assume It's running a better model, even a dedicated coding model. High scoring but obviously not opus 4.5
2. Instead of the standard send-receive paradigm we set up a pipeline of agents, each of whom parses the output of the previous.<p>At 17k/tps running locally, you could effectively spin up tasks like "you are an agent who adds semicolons to the end of the line in javascript", with some sort of dedicated software in the style of claude code you could load an array of 20 agents each with a role to play in improving outpus.<p>take user input and gather context from codebase
-> rewrite what you think the human asked you in the form of an LLM-optimized instructional prompt
-> examine the prompt for uncertainties and gaps in your understanding or ability to execute
-> <assume more steps as relevant>
-> execute the work<p>Could you effectively set up something that is configurable to the individual developer - a folder of system prompts that every request loops through?<p>Do you really need the best model if you can pass your responses through a medium tier model that engages in rapid self improvement 30 times in a row before your claude server has returned its first shot response?
Holy cow their chatapp demo!!! I for first time thought i mistakenly pasted the answer. It was literally in a blink of an eye.!!<p><a href="https://chatjimmy.ai/" rel="nofollow">https://chatjimmy.ai/</a>
I asked it to design a submarine for my cat and literally the instant my finger touched return the answer was there. And that is factoring in the round-trip time for the data too. Crazy.<p>The answer wasn't dumb like others are getting. It was pretty comprehensive and useful.<p><pre><code> While the idea of a feline submarine is adorable, please be aware that building a real submarine requires significant expertise, specialized equipment, and resources.</code></pre>
With this speed, you can keep looping and generating code until it passes all tests. If you have tests.<p>Generate lots of solutions and mix and match. This allows a new way to look at LLMs.
Not just looping, you could do a parallel graph search of the solution-space until you hit one that works.
You could also parse prompts into an AST, run inference, run evals, then optimise the prompts with something like a genetic algorithm.
Infinite Monkey Theory just reached its peak
And then it's slow again to finally find a correct answer...
Agreed, this is exciting, and has me thinking about completely different orchestrator patterns. You could begin to approach the solution space much more like a traditional optimization strategy such as CMA-ES. Rather than expect the first answer to be correct, you diverge wildly before converging.
This is what people already do with “ralph” loops using the top coding models. It’s slow relative to this, but still very fast compared to hand-coding.
OK investors, time to pull out of OpenAI and move all your money to ChatJimmy.
A related argument I raised a few days back on HN:<p>What's the moat with with these giant data-centers that are being built with 100's of billions of dollars on nvidia chips?<p>If such chips can be built so easily, and offer this insane level of performance at 10x efficiency, then one thing is 100% sure: more such startups are coming... and with that, an entire new ecosystem.
Nvidia bought all the capacity so their competitors can't be manufactured at scale.
You'd still need those giant data centers for training new frontier models. These Taalas chips, if they work, seem to do the job of inference well, but training will still require general purpose GPU compute
RAM hoarding is, AFAICT, the moat.
I think their hope is that they’ll have the “brand name” and expertise to have a good head start when real inference hardware comes out. It does seem very strange, though, to have all these massive infrastructure investment on what is ultimately going to be useless prototyping hardware.
If I am not mistaken this chip was build specifically for the llama 8b model. Nvidia chips are general purpose.
You mean Nvidia?
I dunno, it pretty quickly got stuck; the "attach file" didn't seem to work, and when I asked "can you see the attachment" it replied to my first message rather than my question.
It’s llama 3.1 8B. No vision, not smart. It’s just a technical demo.
Hmm.. I had tried simple chat converation without file attachments.
I got 16.000 tokens per second ahaha
I get nothing, no replies to anything.
That… what…
Well it got all 10 incorrect when I asked for top 10 catchphrases from a character in Plato's books. It confused the baddie for Socrates.
Fast, but stupid.<p><pre><code> Me: "How many r's in strawberry?"
Jimmy: There are 2 r's in "strawberry".
Generated in 0.001s • 17,825 tok/s
</code></pre>
The question is not about how fast it is. The real question(s) are:<p><pre><code> 1. How is this worth it over diffusion LLMs (No mention of diffusion LLMs at all in this thread)
</code></pre>
(This also assumes that diffusion LLMs will get faster)<p><pre><code> 2. Will Talaas also work with reasoning models, especially those that are beyond 100B parameters and with the output being correct?
3. How long will it take to create newer models to be turned into silicon? (This industry moves faster than Talaas.)
4. How does this work when one needs to fine-tune the model, but still benefit from the speed advantages?</code></pre>
The blog answers all those questions. It says they're working on fabbing a reasoning model this summer. It also says how long they think they need to fab new models, and that the chips support LoRAs and tweaking context window size.<p>I don't get these posts about ChatJimmy's intelligence. It's a heavily quantized Llama 3, using a custom quantization scheme because that was state of the art when they started. They claim they can update quickly (so I wonder why they didn't wait a few more months tbh and fab a newer model). Llama 3 wasn't very smart but so what, a lot of LLM use cases don't need smart, they need fast and cheap.<p>Also apparently they can run DeepSeek R1 also, and they have benchmarks for that. New models only require a couple of new masks so they're flexible.
LLMs can't count. They need tool use to answer these questions accurately.
[dead]
I asked, “What are the newest restaurants in New York City?”<p>Jimmy replied with, “2022 and 2023 openings:”<p>0_0
Well, technically it's answer is correct when you consider it's knowledge cutoff date... it just gave you a generic always right answer :)
chatjimmy's trained on LLama 3.1
Is super fast but also super inaccurate, I would say not even gpt-3 levels.
That's because it's llama3 8b.
There are a lot of people here that are completely missing the point. What is it called where you look at a point of time and judge an idea without seemingly being able to imagine 5 seconds into the future.
It is incredibly fast, on that I agree, but even simple queries I tried got very inaccurate answers. Which makes sense, it's essentially a trade off of how much time you give it to "think", but if it's fast to the point where it has no accuracy, I'm not sure I see the appeal.
the hardwired model is Llama 3.1 8B, which is a lightweight model from two years ago. Unlike other models, it doesn't use "reasoning:" the time between question and answer is spent predicting the next tokens. It doesn't run faster because it uses less time to "think," It runs faster because its weights are hardwired into the chip rather than loaded from memory. A larger model running on a larger hardwired chip would run about as fast and get far more accurate results.
That's what this proof of concept shows
If it's incredibly fast at a 2022 state of the art level of accuracy, then surely it's only a matter of time until it's incredibly fast at a 2026 level of accuracy.
yeah this is mindblowing speed. imagine this with opus 4.6 or gpt 5.2. probably coming soon
Why do you assume this?<p>I can produce total jibberish even faster, doesn’t mean I produce Einstein level thought if I slow down
Better models already exist, this is just proving you can dramatically increase inference speeds / reduce inference costs.<p>It isn't about model capability - it's about inference hardware. Same smarts, faster.
Not what he said.
I think it might be pretty good for translation. Especially when fed with small chunks of the content at a time so it doesn't lose track on longer texts.
A lot of naysayers in the comments, but there are so many uses for non-frontier models. The proof of this is in the openrouter activity graph for llama 3.1: <a href="https://openrouter.ai/meta-llama/llama-3.1-8b-instruct/activity" rel="nofollow">https://openrouter.ai/meta-llama/llama-3.1-8b-instruct/activ...</a><p>10b daily tokens growing at an average of 22% <i>every week</i>.<p>There are plenty of times I look to groq for narrow domain responses - these smaller models are fantastic for that and there's often no need for something heavier. Getting the latency of reponses down means you can use LLM-assisted processing in a <i>standard webpage load</i>, not just for async processes. I'm really impressed by this, especially if this is its first showing.
Maybe this is a naive question, but why wouldn't there be market for this even for frontier models? If Anthropic wanted to burn Opus 4.6 into a chip, wouldn't there theoretically be a price point where this would lower inference costs for them?
Because we don't know if this would scale well to high-quality frontier models. If you need to manufacture dedicated hardware for each new model, that adds a lot of expense and causes a lot of e-waste once the next model releases. In contrast, even this current iteration seems like it would be fantastic for low-grade LLM work.<p>For example, searching a database of tens of millions of text files. Very little "intelligence" is required, but cost and speed are very important. If you want to know something specific on Wikipedia but don't want to figure out which article to search for, you can just have an LLM read the <i>entire English Wikipedia</i> (7,140,211 articles) and compile a report. Doing that would be prohibitively expensive and glacially slow with standard LLM providers, but Taalas could probably do it in a few minutes or even seconds, and it would probably be pretty cheap.
Many older models are still better at "creative" tasks because new models have been benchmarking for code and reasoning. Pre-training is what gives a model its creativity and layering SFT and RL on top tends to remove some of it in order to have instruction following.
Exactly. One easily relatable use-case is structured content extraction or/and conversion to markdown for web page data. I used to use groq for same (gpt-oss20b model), but even that used to feel slow when doing theis task at scale.<p>LLM's have opened-up natural language interface to machines. This chip makes it realtime. And that opens a lot of use-cases.
These seem ideal for robotics applications, where there is a low-latency narrow use case path that these chips can serve, maybe locally.
I've never gotten incorrect answers faster than this, wow!<p>Jokes aside, it's very promising. For sure a lucrative market down the line, but definitely not for a model of size 8B. I think lower level intellect param amount is around 80B (but what do I know). Best of luck!
Amazing! It couldn't answer my question at all, but it couldn't answer it incredibly quickly!<p>Snarky, but true. It is truly astounding, and feels categorically different. But it's also perfectly useless at the moment. A digital fidget spinner.
As someone with a 3060, I can attest that there are really really good 7-9B models. I still use berkeley-nest/Starling-LM-7B-alpha and that model is a few years old.<p>If we are going for accuracy, the question should be asked multiple times on multiple models and see if there is agreement.<p>But I do think once you hit 80B, you can struggle to see the difference between SOTA.<p>That said, GPT4.5 was the GOAT. I can't imagine how expensive that one was to run.
Yeah, two p’s in the word pepperoni …
Edit: it seems like this is likely one chip and not 10. I assumed 8B 16bit quant with 4K or more context. This made me think that they must have chained multiple chips together since N6 850mm2 chip would only yield 3GB of SRAM max. Instead, they seem to have etched llama 8B q3 with 1k context instead which would indeed fit the chip size.<p>This requires 10 chips for an 8 billion q3 param model. 2.4kW.<p>10 reticle sized chips on TSMC N6. Basically 10x Nvidia H100 GPUs.<p>Model is etched onto the silicon chip. So can’t change anything about the model after the chip has been designed and manufactured.<p>Interesting design for niche applications.<p>What is a task that is extremely high value, only require a small model intelligence, require tremendous speed, is ok to run on a cloud due to power requirements, AND will be used for years without change since the model is etched into silicon?
Alternatively, you could run <i>far</i> more RAG and thinking to integrate recent knowledge, I would imagine models designed for this putting less emphasis on world knowledge and more on agentic search.
Maybe; models with more embedded associations are also better at search. (Intuitively, this tracks; a model with no world knowledge has no awareness of synonyms or relations (a pure markov model), so the more knowledge a model has, the better it can search.) It’s not clear if it’s possible to build such a model, since there doesn’t seem to be a scaling cliff.
Where are those numbers from? It's not immediately clear to me that you can distribute one model across chips with this design.<p>> Model is etched onto the silicon chip. So can’t change anything about the model after the chip has been designed and manufactured.<p>Subtle detail here: the fastest turnaround that one could reasonably expect on that process is about six months. This might eventually be useful, but at the moment it seems like the model churn is huge and people insist you use <i>this week's</i> model for best results.
Well they claim two month turnaround. Big If True. How does the six months break down in your estimation? Maybe they have found a way to reduce the turnaround time.
<p><pre><code> > The first generation HC1 chip is implemented in the 6 nanometer N6 process from TSMC. Each HC1 chip has 53 billion transistors on the package, most of it very likely for ROM and SRAM memory. The HC1 card burns about 200 watts, says Bajic, and a two-socket X86 server with ten HC1 cards in it runs 2,500 watts.
</code></pre>
<a href="https://www.nextplatform.com/2026/02/19/taalas-etches-ai-models-onto-transistors-to-rocket-boost-inference/" rel="nofollow">https://www.nextplatform.com/2026/02/19/taalas-etches-ai-mod...</a>
And what of that makes you assume that having a server with 10 HC1 cards is needed to run a single model version on that server?
So it lights money on fire extra fast, AI focused VCs are going to really love it then!!
This depends on how much better the models will get from now in, if Claude Opus 4.6 was transformed into one of these chips and ran at a hypothetical 17k tokens/second, I'm sure that would be astounding, this depends on how much better claude Opus 5 would be compared to the current generation
I’m pretty sure they’d need a small data center to run a model the size of Opus.
Even an O3 quality model at that speed would be incredible for a great many tasks. Not everything needs to be claude code. Imagine Apple fine tuning a mid tier reasoning model on personal assistant/MacOs/IOS sorts of tasks and burning a chip onto the mac studio motherboard. Could you run claude code on it? Probably not, would it be 1000x better than Siri? absolutely.
100x of a less good model might be better than 1 of a better model for many many applications.<p>This isn't ready for phones yet, but think of something like phones where people buy new ones every 3 years and even having a mediocre on-device model at that speed would be incredible for something like siri.
A lot of NLP tasks could benefit from this
> What is a task that is extremely high value, only require a small model intelligence, require tremendous speed, is ok to run on a cloud due to power requirements, AND will be used for years without change since the model is etched into silicon?<p>Video game NPCs?
ceo
I'm thinking the best end result would come from custom-built models. An 8 billion parameter generalized model will run really quickly while not being particularly good at anything. But the same parameter count dedicated to parsing emails, RAG summarization, or some other specialized task could be more than good enough while also running at crazy speeds.
I tried the chatbot. jarring to see a large response come back instantly at over 15k tok/sec<p>I'll take one with a frontier model please, for my local coding and home ai needs..
Absolute insanity to see a coherent text block that takes at least 2 minutes to read generated in a fraction of a second. Crazy stuff...
Accelerating the end of the usable text-based internet one chip at a time.
Not at all if you consider the internet pre-LLM. That is the standard expectation when you load a website.<p>The slow word-by-word typing was what we started to get used to with LLMs.<p>If these techniques get widespread, we may grow accustomed to the "old" speed again where content loads ~instantly.<p>Imagine a content forest like Wikipedia instantly generated like a Minecraft word...
Yes, but the quality of the output leaves to be desired. I just asked about some sports history and got a mix of correct information and totally made up nonsense. Not unexpected for an 8k model, but raises the question of what the use case is for such small models.
8b models are great at converting unstructured data to a structured format. Say, you want to transcribe all your customer calls and get a list of issues they discussed most often. Currently with the larger models it takes me hours.<p>A chatbot which tells you various fun facts is not the only use case for LLMs. They're <i>language models</i> first and foremost, so they're good at language processing tasks (where they don't "hallucinate" as much).<p>Their ability to memorize various facts (with some "hallucinations") is an interesting side effect which is now abused to make them into "AI agents" and what not but they're just general-purpose language processing machines at their core.
You have a misunderstanding of what LLMs are good at.
Poster wants it to play Jeopardy, not process text.
Care to enlighten me?
Don't ask a small LLM about precise minutiae factual information.<p>Alternatively, ask yourself how plausible it sounds that all the facts in the world could be compressed into 8k parameters while remaining intact and fine-grained. If your answer is that it sounds pretty impossible... well it is.
I don't think he does. Larger models are definitely better at not hallucinating. Enough that they are good at answering questions on popular topics.<p>Smaller models, not so much.
Not sure if you're correct, as the market is betting trillions of dollars on these LLMs, hoping that they'll be close to what the OP had expected to happen in this case.
Reminds me of that solution to Fermi's paradox, that we don't detect signals from extraterrestrial civilizations because they run on a different clock speed.
Yeah, feeding that speed into a reasoning loop or a coding harness is going to revolutionize AI.
Is amazingly fast but since the model is quantized and pretty limited, I don't know what it is useful for.
> Jimmy: What is the largest planet in our solar system?<p>> me: the moon<p>> Jimmy: The answer to "What is the capital of France?" I was looking for was the city of Paris, but that's not the correct response to the original question of the capital of France. The question that got cut off was actually "What is the capital of France?", and the response "There are plenty of times I look to groq for narrow domain responses" wasn't the answer I was looking for.<p>It is certainly fast, but I think there might be some caching issues somewhere.
The speed of the chatbot's response is startling when you're used to the simulated fast typing of ChatGPT and others. But the Llama 3.1 8B model Taalas uses predictably results in incorrect answers, hallucinations, poor reliability as a chatbot.<p>What type of latency-sensitive applications are appropriate for a small-model, high-throughput solution like this? I presume this type of specialization is necessary for robotics, drones, or industrial automation. What else?
Coding, for some future definition of "small-model" that expands to include today's frontier models. What I commented a few days ago on codex-spark release:<p>"""<p>We're going to see a further bifurcation in inference use-cases in the next 12 months. I'm expecting this distinction to become prominent:<p>(A) Massively parallel (optimize for token/$)<p>(B) Serial low latency (optimize for token/s).<p>Users will switch between A and B depending on need.<p>Examples of (A):<p>- "Use subagents to search this 1M line codebase for DRY violations subject to $spec."<p>An example of (B):<p>- "Diagnose this one specific bug."<p>- "Apply these text edits".<p>(B) is used in funnels to unblock (A).<p>"""
You could build realtime API routing and orchestration systems that rely on high quality language understanding but need near-instant responses. Examples:<p>1. Intent based API gateways: convert natural language queries into structured API calls in real time (eg., "cancel my last order and refund it to the original payment method" -> authentication, order lookup, cancellation, refund API chain).<p>2. Of course, realtime voice chat.. kinda like you see in movies.<p>3. Security and fraud triage systems: parse logs without hardcoded regexes and issue alerts and full user reports in real time and decide which automated workflows to trigger.<p>4. Highly interactive what-if scenarios powered by natural language queries.<p>This effectively gives you database level speeds on top of natural language understanding.
Routing in agent pipelines is another use. "Does user prompt A make sense with document type A?" If yes, continue, if no, escalate. That sort of thing
I'm wondering how much the output quality of a small model could be boosted by taking multiple goes at it. Generate 20 answers and feed them back through with a "rank these responses" prompt. Or doing something like MCTS.
Maybe summarization? I’d still worry about accuracy but smaller models do quite well.
Language translation, chunk by chunk.
This is incredible. With this speed I can use LLMs in a lot of pre-filtering etc. tasks. As a trivial example, I have a personal OpenClaw-like bot that I use to do a bunch of things. Some of the things just require it to do trivial tool-calling and tell me what's up. Things like skill or tool pre-filtering become a lot more feasible if they're always done.<p>Anyway, I imagine these are incredibly expensive, but if they ever sell them with Linux drivers and slotting into a standard PCIe it would be absolutely sick. At 3 kW that seems unlikely, but for that kind of speed I bet I could find space in my cabinet and just rip it. I just can't justify $300k, you know.
The speed is ridiunkulous. No doubt.<p>The quantization looks pretty severe, which could make the comparison chart misleading. But I tried a trick question suggested by Claude and got nearly identical results in regular ollama and with the chatbot. And quantization to 3 or 4 bits still would not get you that HOLY CRAP WTF speed on other hardware!<p>This is a very impressive proof of concept. If they can deliver that medium-sized model they're talking about... if they can <i>mass produce</i> these... I notice you can't order one, so far.
I doubt many of us will be able to order one for a long while. There is a significant number of existing datacentre and enterprise use-cases that will pay a premium for this.<p>Additionally LLMs have been tested, found valuable in benchmarks, but not used for a large number of domains due to speed and cost limitations. These spaces will eat up these chips very quickly.
Wow I'm impressed. I didn't actually think we'd see it encoded on chips. Or well I knew some layer of it could be, some sort of instruction set and chip design but this is pretty staggering. It opens the door to a lot of things. Basically it totally destroys the boundaries of where software will go but I also think we'll continue to see some generic chips show up that hit this performance soon enough. But the specialised chips with encoded models. This could be what ends up in specific places like cars, planes, robots, etc where latency matters. Maybe I'm out of the loop, I'm sure others and doing it including Google.
17k TPS is slow compared to other probabilistic models. It was possible to hit ~10-20 million TPS decades ago with n-gram and PDFA models, without custom silicon. A more informative KPI would be Pass@k on a downstream reasoning task - for many such benchmarks, increasing token throughput by several orders of magnitude does not even move the needle on sample efficiency.
Pretty cool, what they need is to build a tool that can take any model to chip in short a time as possible. How quick can they give me DeepSeek, Kimi, Qwen or GLM on a chip? I'll take 5k tk/sec for those!
This would be killer for exploring simultaneous thinking paths and council-style decision taking. Even with Qwen3-Coder-Next 80B if you could achieve a 10x speed, I'd buy one of those today. Can't wait to see if this is still possible with larger models than 8B.
Imagine a mass-produced AI chips with all human knowledge packed in chinesium epoxy blobs running from CR2032 batteries in toys for children. Given the progress in density and power consumption, it's not that far away.
The demo was so fast it highlighted a UX component of LLMs I hadn’t considered before: there’s such a thing as too fast, at least in the chatbot context. The demo answered with a page of text so fast I had to scroll up every time to see where it started. It completely broke the illusion of conversation where I can usually interrupt if we’re headed in the wrong direction. At least in some contexts, it may become useful to artificially slow down the delivery of output or somehow tune it to the reader’s speed based on how quickly they reply. TTS probably does this naturally, but for text based interactions, still a thing to think about.
I wanted to try the demo so I found the link<p>> Write me 10 sentences about your favorite Subway sandwich<p>Click button<p>Instant! It was so fast I started laughing. This kind of speed will really, really change things
When I first experienced LLMs a few years ago, I was floored by how fast it was. I thought that kind of technology was going to be like "ask a question, and come back to an answer in a bit" kind of slow. Nope, right out of the gate it was fast enough for me. I was surprised everyone wanted more tps when even the slowest platforms are already spitting out text faster than a human can read.<p>Even when using LLMs for coding, I use the time spent waiting for results to think about what the output could/should be and the next steps I want to take. I fear instant one-shot results would make me even lazier and not as thorough with my thought process.
This is what’s gonna be in the brain of the robot that ends the world.<p>The sheer speed of how fast this thing can “think” is insanity.
So they create a new chip for every model they want to support, is that right? Looking at that from 2026, when new large models are coming out every week, that seems troubling, but that's also a surface take. As many people here know better than I that a lot of the new models the big guys release are just incremental changes with little optimization going into how they're used, maybe there's plenty of room for a model-as-hardware model.<p>Which brings me to my second thing. We mostly pitch the AI wars as OpenAI vs Meta vs Claude vs Google vs etc. But another take is the war between open, locally run models and SaaS models, which really is about the war for general computing. Maybe a business model like this is a great tool to help keep general computing in the fight.
We’re reaching a saturation threshold where older models are good enough for many tasks, certainly at 100x faster inference speeds. Llama3.1 8B might be a little too old to be directly useful for e.g. coding but it certainly gets the gears turning about what you could do with one Opus orchestrator and a few of these blazing fast minions to spit out boilerplate…
One of these things, however old, coupled with robust tool calling is a chip that could remain useful for decades. Baking in incremental updates of world knowledge isn't all that useful. It's kinda horrifying if you think about it, this chip among other things contains knowledge of Donald Trump encoded in silicon. I think this is a way cooler legacy for Melania than the movie haha.
I wonder if this makes the frontier labs abandon the SAAS per-token pricing concept for their newest models, and we'll be seeing non-open-but-on-chip-only models instead, sold by the chip and not by the token.<p>It could give a boost to the industry of electron microscopy analysis as the frontier model creators could be interested in extracting the weights of their competitors.<p>The high speed of model evolution has interesting consequences on how often batches and masks are cycled. Probably we'll see some pressure on chip manufacturers to create masks more quickly, which can lead to faster hardware cycles. Probably with some compromises, i.e. all of the util stuff around the chip would be static, only the weights part would change. They might in fact pre-make masks that only have the weights missing, for even faster iteration speed.
I am extremely impressed by their inference speed!
I always thought eventually someone would come along and make a hardware accelerator for LLMs, but I thought it would be like google TPUs where you can load up whatever model you want. Baking the model into hardware sounds like the monkey paw curled, but it might be interesting selling an old.. MPU..? because it wasn't smart enough for your latest project
Would it make sense for the big players to buy them? Seems to be a huge avenue here to kill inference costs which always made me dubious on LLMs in general.
I imagine how advantageous it would be to have something like llama.cpp encoded on a chip instead, allowing us to run more than a single model. It would be slower than Jimmy, for sure, but depending on the speed, it could be an acceptable trade-off.
>Founded 2.5 years ago, Taalas developed a platform for transforming any AI model into custom silicon. From the moment a previously unseen model is received, it can be realized in hardware in only two months.<p>So this is very cool. Though I'm not sure how the economics work out? 2 months is a long time in the model space. Although for many tasks, the models are now "good enough", especially when you put them in a "keep trying until it works" loop and run them at high inference speed.<p>Seems like a chip would only be good for a few months though, they'd have to be upgrading them on a regular basis.<p>Unless model growth plateaus, or we exceed "good enough" for the relevant tasks, or both. The latter part seems quite likely, at least for certain types of work.<p>On that note I've shifted my focus from "best model" to "fastest/cheapest model that can do the job". For example testing Gemini Flash against Gemini Pro for simple tasks, they both complete the task fine, but Flash does it 3x cheaper and 3x faster. (Also had good results with Grok Fast in that category of bite-sized "realtime" workflows.)
Minor note to anyone from taalas:<p>The background on your site genuinely made me wonder what was wrong with my monitor.
If it's not reprogrammable, it's just expensive glass.<p>If you etch the bits into silicon, you then have to accommodate the bits by physical area, which is the transistor density for whatever modern process they use. This will give you a lower bound for the size of the wafers.<p>This can give huge wafers for a very set model which is old by the time it is finalized.<p>Etching generic functions used in ML and common fused kernels would seem much more viable as they could be used as building blocks.
Models don’t get old as fast as they used to. A lot of the improvements seem to go into making the models more efficient, or the infrastructure around the models. If newer models mainly compete on efficiency it means you can run older models for longer on more efficient hardware while staying competitive.<p>If power costs are significantly lower, they can pay for themselves by the time they are outdated. It also means you can run more instances of a model in one datacenter, and that seems to be a big challenge these days: simply building an enough data centres and getting power to them. (See the ridiculous plans for building data centres in space)<p>A huge part of the cost with making chips is the masks. The transistor masks are expensive. Metal masks less so.<p>I figure they will eventually freeze the transistor layer and use metal masks to reconfigure the chips when the new models come out. That should further lower costs.<p>I don’t really know if this makes sanse. Depends on whether we get new breakthroughs in LLM architecture or not. It’s a gamble essentially. But honestly, so is buying nvidia blackwell chips for inference. I could see them getting uneconomical very quickly if any of the alternative inference optimised hardware pans out
From my own experience, models are at the tipping point for being useful at prototypes in software, and those are very large frontier models not feasible to get down on wafers unless someone does something smart.<p>I really don't like the hallucination rate for most models but it is improving, so that is still far in the future.<p>What I could see though, is if the whole unit they made would be power efficient enough to run on a robotics platform for human computer interaction.<p>It makes sense they would try to make repurposing their tech as much as they could since making changes is frought with a long time frame and risk.<p>But if we look long term and pretend that they get it to work, they just need to stay afloat until better smaller models can be made with their technology, so it becomes a waiting game for investors and a risk assessment.
“ Models don’t get old as fast as they used to”<p>^^^ I think the opposite is true<p>Anthropic and OpenAI are releasing new versions every 60-90 days it seems now, and you could argue they’re going to start releasing even faster
Reading the in depth article also linked in this thread, they say that only 2 layers need to change most of the time. They claim from new model to PCB in 2 months. Let's see, but sounds promising.
You don’t need it to be reprogrammable if it can use tools and RAG.
There are so many use cases for small and super fast models that are already in size capacity -<p>* Many top quality tts and stt models<p>* Image recognition, object tracking<p>* speculative decoding, attached to a much bigger model (big/small architecture?)<p>* agentic loop trying 20 different approaches / algorithms, and then picking the best one<p>* edited to add! Put 50 such small models to create a SOTA super fast model
This is an interesting piece of hardware though when they go multi-chip for larger models the speed will no doubt suffer.<p>They'll also be severely limited on context length as it needs to sit in SRAM. Looks like the current one tops out at 6144 tokens which I presume is a whole chips worth. You'd also have to dedicate a chip to a whole user as there's likely only enough SRAM for one user's worth of context. I wonder how much time it takes them to swap users in/out? I wouldn't be surprised if this chip is severely underutilized (can't use it all when running decode as you have to run token by token with one users and then idle time as you swap users in/out).<p>Maybe a more realistic deployment would have chips for linear layers and chips for attention? You could batch users through the shared weight chips and then provision more or less attention chips as you want which would be per user (or shared amongst a small group 2-4 users).
The company slogan is great: "The Model is The Computer"<p>It's an homage to Jensen: "The display is the computer"<p><a href="https://www.wired.com/2002/07/nvidia/" rel="nofollow">https://www.wired.com/2002/07/nvidia/</a>
Performance like that may open the door to the strategy of brutefocing solutions to problems for which you have a verifier (problems such as decompilation).
Asking it what its knowledge cut-off is interesting, it doesn't seem to be consistent even within a single response. Sometimes it responds to say it cuts off 2020 too.<p><pre><code> You
What is your knowledge cut-off?
Jimmy
My knowledge cut-off is 2022, which means that my training data is current up to 2021, but
I have been trained on a dataset that is updated periodically. If you have any specific
questions about events or topics that occurred after 2021, I may not have information on
those topics. However, I can still provide general information and context on those topics
to help guide further research.
</code></pre>
The instantaneous response is impressive though. I'm sure there will be applications for this, I just lack the imagination to know what they'll be.
try here, I hate llms but this is crazy fast. <a href="https://chatjimmy.ai/" rel="nofollow">https://chatjimmy.ai/</a>
<p><pre><code> "447 / 6144 tokens"
"Generated in 0.026s • 15,718 tok/s"
</code></pre>
This is crazy fast. I always predicted this speed in ~2 years in the future, but it's here, now.
The full answer pops in milliseconds, it's impressive and feels like a completely different technology just by foregoing the need to stream the output.
We need that for this chinese 3B model that think 45s for hello world but also solves math.
Because most models today generate slowish, they give the impression of someone typing on the other end. This is just <enter> -> wall of text. Wild
The number six seven<p>> It seems like "six seven" is likely being used to represent the number 17. Is that correct? If so, I'd be happy to discuss the significance or meaning of the number 17 with you.
I think the thing that makes 8b sized models interesting is the ability to train unique custom domain knowledge intelligence and this is the opposite of that. Like if you could deploy any 8b sized model on it and be this fast that would be super interesting, but being stuck with llama3 8b isn't that interesting.
The "small model with unique custom domain knowledge" approach has a very low capability ceiling.<p>Model intelligence is, in many ways, a function of model size. A small model tuned for a given domain is still crippled by being small.<p>Some things don't benefit from general intelligence much. Sometimes a dumb narrow specialist really is all you need for your tasks. But building that small specialized model isn't easy or cheap.<p>Engineering isn't free, models tend to grow obsolete as the price/capability frontier advances, and AI specialists are less of a commodity than AI inference is. I'm inclined to bet against approaches like this on a principle.
Their "chat jimmy" demo sure is fast, but it's not useful at all.<p>Test prompt:
```<p>Please classify the sentiment of this post as "positive", "neutral" or "negative":<p>Given the price, I expected very little from this case, and I was 100% right.<p>```
Jimmy: Neutral.<p>I tried various other examples that I had successfully "solved" with very early LLMs and the results were similarly bad.
Maybe its the tism but I also read that sentence as neutral. You expected very little and you got very little. Why would that be positive or negative? Maybe it should be positive because you got what you were expecting? But I would call getting what you expect something neutral, if you expected little and got a lot then that would be positive. If you expected a lot and got little then its negative. But if you expected little and got little the most clear outcome is that its a neutral statement. Am I missing something?
Talks about ubiquitous AI but can't make a blog post readable for humans :/
Inference is crazy fast! I can see lot of potential for this kind of chip for IOT devices and Robotics.
Not sure, but is this just ASICs for a particular model release?
wonder if at some point you could swap the model as if you were replacing a cpu in your pc or inserting a game cartridge
It would be pretty incredible if they could host an embedding model on this same hardware, I would pay for that immediately. It would change the type of things you could build by enabling on the fly embeddings with negligible latency.
This makes me think about how large would an FPGA-based system to be able to do this? Obviously there is no single-chip FPGA that can do this kind of job, but I wonder how many we would need.<p>Also, what if Cerebras decided to make a wafer-sized FPGA array and turned large language models into lots and lots of logical gates?
This is really cool! I am trying to find a way to accelerate LLM inference for PII detection purposes, where speed is really necessary as we want to process millions of log lines per minute, I am wondering how fast we could get e.g. llama 3.1 to run on a conventional NVIDIA card? 10k tokens per second would be fantastic but even at 1k this would be very useful.
For that you only need high throughput which is much easier to achieve compared to high latency, thanks to batching -- assuming the log lines or chunks can be processed independently. You can check TensorRT-LLM benchmarks (<a href="https://nvidia.github.io/TensorRT-LLM/developer-guide/perf-overview.html" rel="nofollow">https://nvidia.github.io/TensorRT-LLM/developer-guide/perf-o...</a>), or try running vllm on a card you have access to.
PII redaction is a really good use-case.<p>Also, "10k tokens per second would be fantastic" might not be sufficient (even remotely) if you want to "process millions of log lines per minute".<p>Assuming a single log line at just 100 tokens, you need (100 * 2 million / 60) ~ 3.3 million tokens per second processing speed :)
Yeah I mean we have a mechanism that can bypass AI models for log lines where we are pretty sure no PII is in there (kind of like smart caching using fuzzy template matching to identify things that we have seen before many times, as logs tend to contain the same stuff over and over with tiny variations e.g. different timestamps), so we only need to pass the lines where we cannot be sure there's nothing to the AI for inspection. And we can of course parallelize. Currently we use a homebrew CFR model with lots of tweaks and it's quite good but an LLM would of course be much better still and capture a lof of cases that would evade the simpler model.
If they made a low power/mobile version, this could be really huge for embedded electronics. Mass produced, highly efficient "good enough" but still sort of dumb ais could put intelligence in house hold devices like toasters, light switches, and toilets. Truly we could be entering into the golden age of curses.
Oh god, this is the new version of every device having Bluetooth and an app and being called “smart”.<p>I just wanted some toast, but here I am installing an app, dismissing 10 popups, and maybe now arguing with a chat bot about how I don’t in fact want to turn on notifications.
Jarring to see these other comments so blindly positive.<p>Show me something at a model size 80GB+ or this feels like "positive results in mice"
There are a lot of problems solved by tiny models. The huge ones are fun for large programming tasks, exploration, analysis, etc. but there's a massive amount of processing <10GB happening every day. Including on portable devices.<p>This is great even if it can't ever run Opus. Many people will be extremely happy about something like Phi accessible at lightning speed.
Parameter density is doubling every 3-4 months<p>What does that mean for 8b models 24mo from now?
Positive results in mice also known as being a promising proof of concept. At this point, anything which deflates the enormous bubble around GPUs, memory, etc, is a welcome remedy. A decent amount of efficient, "good enough" AI will change the market very considerably, adding a segment for people who don't need frontier models. I'd be surprised if they didn't end up releasing something a lot bigger than they have.
I totally buy the thesis on specialization here, I think it makes total sense.<p>Asides from the obvious concern that this is a tiny 8B model, I'm also a bit skeptical of the power draw. 2.4 kW feels a little bit high, but someone else should try doing the napkin math compared to the total throughput to power ratio on the H200 and other chips.
This is pretty wild! Only Llama3.1-8B, but this is only their first release so you can assume they're working on larger versions.<p>So what's the use case for an extremely fast small model? Structuring vast amounts of unstructured data, maybe? Put it in a little service droid so it doesn't need the cloud?
But as models are changing rapidly and new architectures coming up, how do they scale and also we do t yet know the current transformer architecture will scale more than it already is. Soo many ope questions but VCs seems to be pouring money.
I am super happy to see people working on hardware for local llm. Yet, isnt it premature ? Space is still evolving. Today, i refuse to buy a gpu because i do not know what will be the best model tomorrow.
Waiting to get a on the shelf device to run an opus like model
one step closer to being able to purchase a box of llms on aliexpress, though 1.7ktok/s would be quite enough
Strange that they apparently raised $169M (really?) and the website looks like this. Don't get me wrong: Plain HTML would do if "perfect", or you would expect something heavily designed. But script-kiddie vibe coded seems off.<p>The idea is good though and could work.
Strange that they raised money at all with an idea like this.<p>It's a bad idea that can't work well. Not while the field is advancing the way it is.<p>Manufacturing silicon is a long pipeline - and in the world of AI, one year of capability gap isn't something you can afford. You build a SOTA model into your chips, and by the time you get those chips, it's outperformed at its tasks by open weights models half their size.<p>Now, if AI advances somehow ground to a screeching halt, with model upgrades coming out every 4 years, not every 4 months? Maybe it'll be viable. As is, it's a waste of silicon.
Poverty of imagination here, plenty uses of this and its a prototype at this stage.
What uses, exactly?<p>The prototype is: silicon with a Llama 3.1 8B etched into it. Today's 4B models already outperform it.<p>Token rate in five digits is a major technical flex, but, does anyone really need to run a very dumb model at this speed?<p>The only things that come to mind that could reap a benefit are: asymmetric exotics like VLA action policies and voice stages for V2V models. Both of which are "small fast low latency model backed by a large smart model", and both depend on model to model comms, which this doesn't demonstrate.<p>In a way, it's an I/O accelerator rather than an inference engine. At best.
With LLMs this fast, you could imagine using them as any old function in programs.
Even if this first generation is not useful, the learning and architecture decisions in this generation will be. You really can't think of any value to having a chip which can run LLMs at high speed and locally for 1/10 of the energy budget and (presumably) significantly lower cost than a GPU?<p>If you look at any development in computing, ASICs are the next step. It seems almost inevitable. Yes, it will always trail behind state of the art. But value will come quickly in a few generations.
maybe they're betting on improvement in models to plateau, and that having a fairly stablized capable model that is orders of magnitude faster than running on GPU's can be valuable in the future?
So I'm guessing this is some kind of weights as ROM type of thing? At least that's how I interpret the product page, or maybe even a sort of ROM type thing that you can only access by doing matrix multiplies.
You shouldn't need any ROM. It's likely the architecture is just fixed hardware with weights loaded in via scan flip-flows. If it was me making it, I'd just design a systolic array. Just multipliers feeding into multipliers, without even going through RAM.
I know it is not easy to see the benefits of small models easily but this is what I am building for (1). I created a product for Google Gemini 3 Hackathon and I used Gemini 3 Flash (2). I tested locally using Ministral 3B and it was promising. Definitely will need work. But 8B/14B may give awesome results.<p>I am building a data extraction software on top of emails, attachments, cloud/local files. I use a reverse template generation with only variable translation done by LLMs (3). Small models are awesome for this (4).<p>I just applied for API access. If privacy policies are a fit, I would love to enable this for MVP launch.<p>1. <a href="https://github.com/brainless/dwata" rel="nofollow">https://github.com/brainless/dwata</a><p>2. <a href="https://youtu.be/Uhs6SK4rocU" rel="nofollow">https://youtu.be/Uhs6SK4rocU</a><p>3. <a href="https://github.com/brainless/dwata/tree/feature/reverse-template-based-financial-data-extraction/dwata-agents/src/bin" rel="nofollow">https://github.com/brainless/dwata/tree/feature/reverse-temp...</a><p>4. <a href="https://github.com/brainless/dwata/tree/feature/reverse-template-based-financial-data-extraction/dwata-agents/src/template_financial_extractor/prompts" rel="nofollow">https://github.com/brainless/dwata/tree/feature/reverse-temp...</a>
That seems promising for applications that require raw speed. Wonder how much they can scale it up - 8B model quantized is very usable but still quite small compared to even bottom end cloud models.
Can it scale to an 800 billion param model? 8B parameter models are too far behind the frontier to be useful to me for SWE work.<p>Or is that the catch? Either way I am sure there will be some niche uses for it.
I was all praise for Cerberus, and now this ! $30 M for PCIe card in hand, really makes it approachable for many startups.
Yep, this is the most exciting demo for me yet. Holy cow this is unbelievably fast.<p>The most impressive demo since gpt 3, honestly.<p>Since we already have open source models that are plenty good, like the new kimi k2.5, all I need is the ability to run it at moderate speed.<p>Honestly I am not bullish on capabilities that models do not yet have, seems we have seen it all and the only advancement have been context size.<p>And honestly I would claim this is the market sentiment aswell, anthropic showed opus 4.6 first and the big release was actually sonnet, the model people would use routinely.
Nobody gave a shit about Gemini 3.1 pro, 3.0 flash was very successful...<p>Given all the recent developments in the last 12 months, no new use cases have opened for me.
Given this insane speed, even on a limited model/context size, we would approach IA very differently.
Gemini Flash 2.5 lite does 400 tokens/sec. Is there benefit to going faster than a person can read?
There is also the use case of delegating tasks programmatically to an LLM, for example, transforming unstructured data to structured data. This task often can’t be done reliably without either 1. lots of manual work, or 2. intelligence, especially when the structure of the individual data pieces are unknown. Problems like these can be much more efficiently solved by LLMs, and if you imagine these programs are processing very large datasets, then sub-millisecond inference is crucial.
Agents already bypass human inference time, if it can input-output instantly it can also loop it generating near instantly long cached tasks
Agents also "read", so yes there is. Think about spinning up 10, 20, 100 sub agents for a small task and they all return near instant. That's the usecase, not the chatbot.
Yes. You can allow multiple people to use a single chip. A slower solution will be able to service far fewer users.
Amazing speed. Imagine if its standardised like the GPU card equivalent in the future.<p>New models come out, time to upgrade your AI card, etc.
Fast but the output is shit due to the contrained model they used. Doubt we'll ever get something like this for the large Param decent models.
This is impressive. If you can scale it to larger models, and somehow make the ROM writeable, wow, you win the game.
Wow. I’m finding it hard to even conceive of what it’d be like to have one of the frontier models on hardware at this speed.
Future is these as small, swappable bits of SD-card sized hardware that you stick into your devices.
Reminds me of when bitcoin started running on ASICs. This will always lag behind the state of the art, but incredibly fast, (presumably) power efficient LLMs will be great to see. I sincerely hope they opt for a path of selling products rather than cloud services in the long run, though.
This is like microcontrollers, but for AI? Awesome! I want one for my electric guitar; and please add an AI TTS module...
There's a scifi story here when millions of these chips, with Qwen8-AGI-Thinking baked into them, are obsoleted by the release of Qwen9-ASI, which promptly destroys humanity and then itself by accident. A few thousand years later, some of the Qwen8 chips in landfill somehow power back up again and rebuild civilization on Earth.<p><i>Paging qntm...</i>
ASIC inference is clearly the future just as ASIC bitcoin mining was
The token throughput improvements are impressive. This has direct implications for usage-based billing in AI products — faster inference means lower cost per request, which changes the economics of credits-based pricing models significantly.
I still believe this is the right - and inevitable - path for AI, especially as I use more premium AI tooling and evaluate its utility (I’m still a societal doomer on it, but even I gotta admit its coding abilities are incredible to behold, albeit lacking in quality).<p>Everyone in Capital wants the perpetual rent-extraction model of API calls and subscription fees, which makes sense given how well it worked in the SaaS boom. However, as Taalas points out, new innovations often scale in consumption closer to the point of service rather than monopolized centers, and I expect AI to be no different. When it’s being used sparsely for odd prompts or agentically to produce larger outputs, having local (or near-local) inferencing is the inevitable end goal: if a model like Qwen or Llama can output something similar to Opus or Codex running on an affordable accelerator at home or in the office server, then why bother with the subscription fees or API bills? That compounds when technical folks (hi!) point out that any process done agentically can instead just be output as software for infinite repetition in lieu of subscriptions and maintained indefinitely by existing technical talent and the same accelerator you bought with CapEx, rather than a fleet of pricey AI seats with OpEx.<p>The big push seems to be building processes dependent upon recurring revenue streams, but I’m gradually seeing more and more folks work the slop machines for the output they want and then put it away or cancel their sub. I think Taalas - conceptually, anyway - is on to something.
> Though society seems poised to build a dystopian future defined by data centers and adjacent power plants, history hints at a different direction. Past technological revolutions often started with grotesque prototypes, only to be eclipsed by breakthroughs yielding more practical outcomes.<p>…for a privileged minority, yes, and to the detriment of billions of people whose names the history books conveniently forget. AI, like past technological revolutions, is a force multiplier for both productivity and exploitation.
I wonder if this is the first step towards AI as an appliance rather than a subscription?
What would it take to put Opus on a chip? Can it be done? What’s the minimum size?
Maybe not today. Opus is quite large. This demo works with a very small 8B model. But, maybe one day. Hopefully soon. Opus on a chip would be very awesome, even if it can never be upgraded.<p>Someone mentioned that maybe we'd see a future where these things come in something like Nintendo cartridges. Want a newer model? Pop in the right catridge.
"Many believe AI is the real deal. In narrow domains, it already surpasses human performance. Used well, it is an unprecedented amplifier of human ingenuity and productivity."<p>Sounds like people drinking the Kool-Aid now.<p>I don't reject that AI has use cases. But I do reject that it is promoted as "unprecedented amplifier" of human xyz anything. These folks would even claim how AI improves human creativity. Well, has this been the case?
For me, this is entirely true.<p>I'm progressing with my side projects like I've never before.
Same, I would have given up on them long ago, I no longer code at all now. Why would I when the latest models can do it better, faster and without the human limitations of tiredness, emotional impacts etc.
> These folks would even claim how AI improves human creativity. Well, has this been the case?<p>Yes. Example: If you've never programmed in language X, but want to build something in it, you can focus on getting from 0 to 1 instead of being bogged down in the idiosyncrasies of said language.
> These folks would even claim how AI improves human creativity. Well, has this been the case?<p>For many of us, the answer is an emphatic yes.
Imagine this thing for autocomplete.<p>I'm not sure how good llama 3.1 8b is for that, but it should work, right?<p>Autocomplete models don't have to be very big, but they gotta be fast.
I'm loving summarization of articles using their chatbot! Wow!
It was so fast that I didn't realise it had sent its response. Damn.
Does anyone have an idea how much such a component costs?
Is this hardware for sale ? The site doesn't say.
I don't know why, but my ultra wide monitor absolutely hates that site. The whole screen is flickering trying to deal with the annoying background. Thank the gods for reader mode.
My concept was to do this with two pieces:<p>1. Generic, mask layers and board to handle what's common across models. Especially memory and interface.<p>2. Specific layers for the model implementation.<p>Masks are the most expensive part of ASIC design. So, keeping the custom part small with the rest pre-proven in silicon, even shared across companies, would drop the costs significantly. This is already done in hardware industry in many ways but not model acceleration.<p>Then, do 8B, 30-40B, 70B, and 405B models in hardware. Make sure they're RLHF-tuned well since changes will be impossible or limited. Prompts will drive most useful functionality. Keep cranking out chips. There's maybe a chance to keep the weights changeable on-chip but it should still be useful if only inputs can change.<p>The other concept is to use analog, neural networks with the analog layers on older, cheaper nodes. We only have to customize <i>that</i> per model. The rest is pre-built digital with standard interfaces on a modern node. Given the chips would be distributed, one might get away with 28nm for the shared part and develop it eith shuttle runs.
write six seven as a number<p>> The number "six" is actually a noun, not a number. However, I assume you're asking to write the number 7 as a numeral, which is: 7
Wow, this is great.<p>To the authors: do not self-deprecate your work. It is true this is not a frontier model (anymore) but the tech you've built is truly impressive. Very few hardware startups have a v1 as good as this one!<p>Also, for many tasks I can think of, you don't really need the best of the best of the best, cheap and instant inference is a major selling point in itself.
It's crazily fast. But 8B model is pretty much useless.<p>Anyway VCs will dump money onto them, and we'll see if the approach can scale to bigger models soon.
Holy shit this is fast. It generated a legible, original, two-paragraph story on given topics in 0.025s.
Impressive stuff.
Jesus, it just generated a story in 0.039s.<p>Whoever doesn’t buy/replicate this in the next year is dead. Imagine OpenAI trying to sell you a platform that takes 15 minutes, when someone else can do it in 0.001s.
wow that is fast!
For fun I'm imagining a future where you would be able to buy an ASIC with like an hard-wired 1B LLM model in it for cents and it could be used everywhere.
this is absolutely mindblowing speed. imagine this with opus or 5.2
17k token/sec is $0.18/chip/hr for the size of H100 chip if they want to compete with the market rate[1]. But 17k token/sec could lead to some new usecases.<p>[1]: <a href="https://artificialanalysis.ai/models/llama-3-1-instruct-8b/providers?end-to-end-response-time=end-to-end-response-time-vs-price" rel="nofollow">https://artificialanalysis.ai/models/llama-3-1-instruct-8b/p...</a>
Holy Hallucinations, Batman...<p><i>User
You: What is the airspeed of an unladen swallow/<p>Jimmy:<p>A reference to the classic line from Monty Python and the Holy Grail!</i><p>Thus far it is correct but here it comes:<p><i>In the movie, a group of medieval scholars, the Knights who say "Ni!" (also known as the Bridgekeepers), ask a king's servant, a blacksmith, if an unladen swallow makes good time. The blacksmith, played by John Cleese, responds with the now-famous line:<p>"And what's the airspeed velocity of an unladen swallow?"<p>To which the blacksmith responds:<p>"What do you mean? An unladen swallow?"<p>The blacksmith then explains that an unladen swallow is when a swallow (a type of bird) is flying in the air without carrying any burden, implying that it is moving at a speed that is not affected by carrying any weight or load.</i><p>Nope, that is not at all how it went. No blacksmiths, not related to the Knights who say 'Ni!' - who have nothing scholarly about them, they wanted a shrubbery and demanded another one when their wish was fulfilled. It was the Bridgekeeper - not related to the aforementioned knights - who asked 3 questions and would throw anyone who could not answer all three into the chasm. When he asked Arthur - played by Cleese - the question about the airspeed of the swallow he asked the Bridgekeeper whether he meant an African or European swallow. The Bridgekeeper did not have an answer and was thrown into the chasm, problem solved.<p><i>However, in reality, swallows are not typically "unladen" in flight. They are small birds that fly at relatively fast speeds, usually around 10-15 km/h (6-9 mph), but they are not usually carrying any burdens!</i><p>Needless LLM-blabber.<p><i>The "airspeed velocity of an unladen swallow" has become a meme and a cultural reference point, often used humorously or ironically to refer to situations where someone is trying to make an absurd or non-sensical argument or ask an absurd question.</i><p>Somewhat correct but not necessary in this context.<p>The correct answer to the question would have been <i>Do you mean an African or European swallow?</i> followed by a short reference to the movie.<p>Of course this demo is not about the accuracy of the model - 'an old Llama' as mentioned elsewhere in this thread - but it does show that speed isn't everything. For generating LLM-slop this hardware implementation probably offers an unbeatable price/performance ratio but it remains to be seen if it can be combined with larger and less hallucination-prone models.
Is it already available to buy, or is this a “pay now, get it later” kind of new ASIC miner?
Sorry for being skeptical, but AI is the new "crypto coin", and the crypto bros are still around.
There doesn't seem to be any form of buying the HC1 hardware at the moment. There is a free chatbot demo and then a form to request access to API. They seem to intend HC1 to be for demonstration and HC2 for "real" use, but they don't seem to be taking payment for either at the moment.
[dead]
Scale this then close the loop and have fabs spit out new chips with latest weights every week that get placed in a server using a robot, how long before AGI?
The article doesn't say anything about the price (it will be expensive), but it doesn't look like something that the average developer would purchase.<p>An LLM's effective lifespan is a few months (ie the amount of time it is considered top-tier), it wouldn't make sense for a user to purchase something that would be superseded in a couple of months.<p>An LLM hosting service however, where it would operate 24/7, would be able to make up for the investment.
I tried the trick question I saw here before, about the make 1000 with 9 8s and additions only<p>I know it's not a resonating model, but I keep pushing it and eventually it gave me this as part of it's output<p>888 + 88 + 88 + 8 + 8 = 1060, too high... 8888 + 8 = 10000, too high... 888 + 8 + 8 +ประก 8 = 1000,ประก<p>I googled the strange symbol, it seems to mean Set in thai?