There's a really interesting trend of labs using "proprietary" methods to convert existing models to a compressed ternary format.<p>PrismML actually targeted the same Qwen 8b model and got it down to 1.75gb here: <a href="https://prismml.com/news/ternary-bonsai" rel="nofollow">https://prismml.com/news/ternary-bonsai</a><p>I wonder how proprietary it all is though, since the BitNet b1.58 paper has been out for a couple years now: <a href="https://arxiv.org/abs/2402.17764" rel="nofollow">https://arxiv.org/abs/2402.17764</a><p>From the wikipedia on 1.58 bit llms: "BitNet derives its performance from being trained natively in 1.58 bit instead of being quantized from a full-precision model after training. Still, training is an expensive process, and it would be desirable to be able to somehow convert an existing model to 1.58 bits. In 2024, HuggingFace reported a way to gradually ramp up the 1.58-bit quantization in fine-tuning an existing model down to 1.58 bits."<p>The section from huggingface is here: <a href="https://huggingface.co/blog/1_58_llm_extreme_quantization#fine-tuning-in-158bit" rel="nofollow">https://huggingface.co/blog/1_58_llm_extreme_quantization#fi...</a><p>I just wonder how many of these labs are basically following the huggingface recipe here and possibly tweaking it and releasing models without huge training costs.
The content on that page is too AI-generated to make sense to me; I don't understand what the model is for.
I have the same question.<p>I get that it’s designed to run on a CPU, big GPU or MacBook (although the way that was phrased confused me at first).<p>I’m struggling with what a “decoder-only” model is good for.
AI doesn't want anything, so it doesn't care whether it conveys meaning in its writing. And, apparently the developers of this project also don't care whether it conveys meaning. They just assume we'll wade through the slop? I dunno.
There is not a single person mentioned on the website, github created 3 days ago, no real contact, everything hidden. Completely anonymous. Domain owner hidden.
Can’t say I’m a fan of containers for this. A big chunk of local LLM gains come (imo) from the open modular nature of llama.cpp and friends. Easy to modify. Easy to experiment.<p>Containers are the proprietary binary blob in hardware world equivalent
What? How are those even related with each other? You can just as easy modify and experiment with llama.cpp in a container as outside of it, they really shouldn't impact one another. Containers don't suddenly make llama.cpp less "open modular" somehow, and I'm not sure how you'd arrive as such conclusion.
Largely outperformed by Ternary-Bonsai-8B by their own chart, doesn't seem clear what their special sauce is here.
So I'd like to see a Nemotron3 Ultra converted to a 1.58bit format then have them retrain on the open dataset.
So... not a new subatomic particle discovery then.
Unfortunately it crashed out 'no space left on device' while installing the python demo/quickstart.<p>Only problem was there is plenty of space on the device. PLENTY (not quite 750gb).
There’s a new announcement every other day wrt models. How do y’all keep track of them all same know what’s decent? Good grief!<p>And if it’s decent today, it’s shit in eight months! I tool hop as much as the next dev but this is a bit much.
I just tune out. It’s not worth knowing, following every development in the field. If something works now it will probably work in 8 months even if it’s no longer the new hype thing. Who cares.<p>Not using any of it is also a valid option though it doesn’t satisfy your FOMO. But nothing ever will.
There's no need to, the 50 foot view is simply that many alternatives exist and they mostly fall into 3 meaningful weight classes with comparable performance among each class's members: too expensive to use indiscriminately, too big to run at home, and too small for complex work. As for names and faces in between, the overarching conclusion is that we're rapidly approaching commodity status and those don't really matter much
The real strength of an 8B model is efficiency. It will be interesting to see the balance between performance and inference cost.
This all sounds like middle-out
AI slop site with AI slop research...<p>Blog populated with incoherent PR material generated by Yet Another AI.<p>Sigh...
Slop article, slop site... slop model?
[flagged]