Bonsai 27B (1-bit LLM): The First 27B-Class Model to Run on a Phone

6 comments

• liuliu31 minutes ago
  The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
  • liuliu21 minutes ago
    You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.
• simonw17 minutes ago
  The models themselves are showing up on Hugging Face here: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;prism-ml&#x2F;models" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;prism-ml&#x2F;models</a>
• xyzsparetimexyz10 minutes ago
  That&#x27;s awesome. What&#x27;s the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?
• alvatech38 minutes ago
  TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
  • NitpickLawyer16 minutes ago
    There&#x27;s two variants of this (or, as the joke goes, for very big values of bit):<p>Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.<p>1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
  • bensyverson28 minutes ago
    Yeah, it&#x27;s an unfortunate convention from the very first &quot;1 bit&quot; model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.
• Havoc18 minutes ago
  This must be some sort of unpublished app?<p>I can just see their image tool on the app store
• ai_fry_ur_brain21 minutes ago
  [dead]