MSA: Memory Sparse Attention

4 comments

• kingstnap1 hour ago
  I do wonder about the usefulness about this massive context dumping exercise. 100M is a ridiculous amount. Usually to get good results on practical tasks you need to actually think about what you are dumping into context.<p>I also have my gripes about the way 2 hop is mentioned here. With figure 3 being the canonical example of what I would consider too trivial&#x2F;misleading (The exact text match of &quot;Eric Watts&quot; being in the question and in the context). It leads to the natural question of how does it do compared to an LLM with a grep tool.<p>What I would consider more interesting is practical synthesis over such a large context where you can&#x27;t just string lookup answers. For example maybe dumping all of Intel&#x27;s x86 manuals into context and then asking an LLM to try to write assembly or something.
  • baq1 hour ago
    100M tokens should be enough to put all but the absolutely biggest code bases into a single context. It’s probably also about as much as a single average person in the West reads in a lifetime (make of that what you will philosophically); all x86 manuals should fit nicely with room to spare.
• cyanydeez3 hours ago
  Neat. Can&#x27;t wait for our language, framework specific tools for models. I don&#x27;t need my models writing shakespeare, unless I&#x27;m working on shakespeare.
• algolint1 hour ago
  [dead]
• mememememememo3 hours ago
  [dead]