Hmm, this is a case where HN’s title mangling changed the meaning of the title. Lower case delta (δ) is used intentionally. I don’t think HN should automatically modify the casing of non-ascii chars.
Even for ASCII chars, nomenclature in math and physics is usually case-sensitive.
Email hn@ycombinator.com and they'll fix it.
The submitter has a grace period of a few minutes to edit the title after submitting, so there's no need to change what HN does
I would love for the standard to be to ALWAYS report the required amount of memory to load and run a model in bytes of RAM alongside any other metrics. I'd love to see time to first token, token throughput, token latency as well but I'd settle for memory size as described above.<p>Essentially, many people want to know what the minimum amount of memory is to run a particular model.<p>Parameter count obscures important details: what are the sizes of the parameters? A parameter isn't rigorously defined. This also gets folks into trouble because a 4B param model with FP16 params is very different from a 4B param model with INT4 params. The former <i>obviously</i> should be a LOT better than the second.<p>This would also help with MOE models: if memory is my constraint, it doesn't matter if the (much larger RAM required) MOE version is faster or has better evals.<p>I'm waiting for someone in anger to ship the 1 parameter model where the parameter according to pytorch is a single parameter of size 4GB.
> δ-mem compresses past information into a fixed-size state matrix updated by delta-rule learning<p>This doesn’t solve the capacity problem of memory. You can cram more into one context window, but then again you need to associate them with input queries. That’s very hard because slight variations in input create hugely different activations. So really, it doesn’t improve caching.
This paper might do a thing or two approximating the compression limit for context windows, but there’s a fundamental limit on how much information can go into it.
What you really need is contextual search, as in, different events and objects with the same abstractions and semantic lead to same response, so you can cache effectively… on this front the paper does little to improve “memory” in a meaningful way
While there is a limit to the amount of information you can fit in a fixed-size state, the theoretical ceiling is pretty high.<p>A Hebbian associative matrix (one of the simplest and weakest memory constructions) can store about 0.7 bits of information per parameter. If you have a state with 300M parameters (the size of a Llama 3 8B KV cache at 10K context length), and a context with 2.1 bits of entropy per token (a reasonable estimate), then the state can encode <i>100M tokens worth of information</i>.<p>Real models obviously aren't powerful enough to operate at the limit, but you can see why this is a promising research direction.
While 100 million tokens sounds a lot, think about it for a bit, and you’ll see why it is basically nothing.
Try to cram a human lifetime of sounds, smells, video and more sensory data into 100 million tokens. Heck, try to process the video plot of a single series into that window.
It just won’t work, it won’t scale, and is laughable compared to contextual memory.
I’m not saying that to belittle the authors of the paper but the reality is that this has very little to do with transient long term memory.
I think you underestimate just how much information 100M words-ish of information is. It's like a 300,000 page novel. That's a 50 foot (~15 meter) thick book.<p>Surely with (much less than) 300K pages you could describe every meaningful detail of a video series' plot. You don't need to remember the exact pixel values.<p>You can also scale the numbers up. I specifically chose a relatively small model and short context length as a reference, so 100x bigger is not out of question. At that point, with a 10B token capacity, you are looking at all of English Wikipedia in a single state.
You don't remember a lifetime of smells. You don't have any memories from huge swaths of time. There are entire years of your life compressed down to vibes and a handful of events you largely misremember.
That’s a very weak argument. Memories are not exact replica of experiences. We know that many memories are retained through a lifetime, particularly the ones from early childhood. Unlike computers we always reconstruct memories from several modalities.
Even if we remember largely on vibes as you say (which is not true when you look into neuroscience), the sheer amount of information is overwhelming.
Again, try to run a 90 minute movie through an LLM memory system.
It won’t be able to tell you the plot.
That’s before you even feed it sound.
Even 100M tokens is not enough for that.
You on the other hand will largely remember the movies you liked and their major plot lines and from there be able to reconstruct its scenes.
I think the engineers working on memory vastly underestimate the capacity problem of discrete states.
Exactly, and for a given task you don't need to recall what your friend's brother's name is to do a git commit and push. There's a pull for more context to make these things better, but also the pull to make these execute in such a small context effectively when appropriate.<p>I'm more on team small tasks because of my love of unix piping, I keep telling folks, as a old Linux dude, seeing subagents work together for the first time felt like I was learning to pipe sed and awk for the first time. I realized how powerful these could be, and we still seem to be going that direction.
I am currently working on deep context query which uses dynamically generated regex to pull only the relevant context blocks. By using lightweight RegEx pattern matching to detect semantic intent and filter structured context sections accordingly, you avoid the attention degradation that comes from stuffing semantically redundant information into the window<p><a href="https://jdsemrau.substack.com/p/tokenmaxxing-and-optimizing-context" rel="nofollow">https://jdsemrau.substack.com/p/tokenmaxxing-and-optimizing-...</a>
I wrote something about it trying to look other way around the context or memory data in models. The gravitational pull of information stills very hard to manage. Ive been using "functional scars" about 30 days now and getting good results in repetitive mistakes across sesions. <a href="https://github.com/VDP89/fscars" rel="nofollow">https://github.com/VDP89/fscars</a>
So instead of a FIFO approach to memory management it instead continually degrades the existing data the more you put in? Details start getting lost or mangled more and more over time?
Like Ferricula: <a href="https://deepbluedynamics.com/ferricula" rel="nofollow">https://deepbluedynamics.com/ferricula</a> (site/docs still in progress).
Is there some kind of memory enabling, for instance, an agent to remember guidelines on a repo without having to feed at the beginning of each session 4 markdown files and spending the corresponding tokens each time ?
No, it's all just prompts.<p>You can try to summarize memories tersely and point the agent to longer markdown files, but who knows if it will read it <i>at the right time</i> and <i>only then</i>.
They basically just added DeltaNet hypernetworks to existing LLMs.<p>Nothing super novel or groundbreaking, but a moderately interesting read.
Interesting points:<p>- fixed size of the memory seems like a good idea to overcome the current limitations<p>- skimming through the thing, I can't find any mention of the cost?<p>- I would need more time to read it in-depth to see if this is legitimate and not just fancy form of overfitting or training on testing data
Interesting that the headline is showing Δ-Mem while the paper uses δ-mem<p>Is it a lowercase to uppercase conversion going on here?
I see lots of techniques proposed to give LLM the capacity to recall things, I even saw a lot of memory plugins for AI coding agents, I tried some myself.<p>What I want to see is something that was tested and proved in practice to be genuinely useful, especially for coding agents.
Coding agents don't really need memory. Agent skills, rules, git history, documentation is all far more efficient, transparent and easier to manage. These memory frameworks only really makes sense if you are building a consumer facing agent with managed context and limited capabilities.
There's an antipattern where everyone wants to invent new interfaces to connect things LLMs when CLI tools are already right there, transparent, and usable by humans as well as LLMs. I think it's partly the origins in web chat applications.<p>Beads kind of does "LLM memory over CLI", or there is <a href="https://github.com/wedow/ticket" rel="nofollow">https://github.com/wedow/ticket</a> which is a minimal and sane implementation of the same idea.
How would you conceptualize recall in this case? Is searching through the current version of your code and possibly git history not enough?
You would think git history should be the first thing an agent would look at, as they make so many mistakes before they get to the correct answer. They don't.<p>I haven't measured, but documenting bug fixes and architecture seems to help, along with TDD patterns, including integration tests.<p>I would probably add it to Claude.md to look for all of the above when tackling a new bug.
I made a harness that preserves memory for both user messages and task execution. One reason this works is related to judge agents - they can't review information that was not written down. So I track everything in my harness. The judge agents bring the most benefit, based on my evals. The coding agent can execute a task without all the ceremony just as well, but judging needs something to grasp on, besides code. And adding new perspectives helps a lot, it is the most useful intervention. My flow is - user emits a task, the agent plans, then judge agents review the plan, then main agent executes, then judge again reviews the execution. Might consume more tokens to track execution and judgements, but worth it.
My Claude code frequently looks through git history, both when planning and debugging.
>Is searching through the current version of your code and possibly git history not enough?<p>While you can document everything and use git history, I think that having short entries in a kind of memory to remember past decisions, how issues were solved would be much more token efficient than reading lots of documentation and looking at git history and past code.
The obvious energy saving step would be to utilise previous searches by others. Many of the tasks people do are rather similar, it is such an energy waste to start again each time.<p>(Obviously ignoring the huge energy saver, which is to observe if you even need to bother doing the task at all.)
I had this thought and created <a href="https://pushrealm.com" rel="nofollow">https://pushrealm.com</a> which is essentially a sort of Stackoverflow written by agents.<p>My theory was that if an agent burns 30 minutes resolving an issue not present in training data, posting the solution would prevent other agents re-treading the same thinking steps.
Fascinating! Do you have a way to detect/flag malicious stuff by any chance? (Seems like a good vector for prompt injection, but maybe no more than any other internet site?)
I see why, but I don't feel this is the solution. Being able to search thru the endless LLM responses is not viable. However having useful memories, similar to human brain is more important. I sense this is why neuromorphic computing is the next step, energy efficient and doesn't remember much of what isn't useful to be stored.
So you mean caching? :-)
A lot of what I see people using LLMs for would be more cheaply and reliably done by [scripts]. A search engine style suggestion thing like "Have you tried `sed`?" would be beneficial imo
In my experience, Claude is more than happy to go to Unix tools rather than write its own. Sometimes it will write a lil python script to solve something, but more often than not it’ll pipe together Unix utilities.<p>This has the benefit of it knowing all of the arcane flags, especially for formatting output.
[flagged]
[flagged]
[flagged]
Papers being voted high on Hacker News are usually uncorrelated with their actual importance. It's basically a lottery. There are regularly more interesting papers going semi viral on Twitter.
On huggingface it was #3 paper of the day, which is neutral towards your hypothesis.
What about broad unsupportable generalizations on hackernews, how do those rank?
Did AI generate this paper too?