Unsloth Dynamic 3.0 GGUFs

4 comments

• xlayn20 minutes ago
  Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.
  • mike-the-brain19 minutes ago
    you can still have it, no?<p>&gt; We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
• mike-the-brain19 minutes ago
  Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks&#x2F;months)?
• throwa35626216 minutes ago
  <p><pre><code> &quot;We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller&quot; </code></pre> This is crazy! But has anyone tried these lower quants on real projects?
  • kennywinker7 minutes ago
    Not 1-bit, but I’m getting pretty good results with some light coding using unsloth’s previous 2-bit quant of qwen3.8-27b. With these new quants i may be able to bump up to 3bit, tho it’s already running so slow (15tok&#x2F;s average for the first 32k of context) that the speed hit might make it not worth the extra smarts
• jadbox10 minutes ago
  The new IQ4XS has been working pretty well so far on 4090 16gb.