LLM memory

When an AI remembers everything, knowing what to forget becomes the real skill

A coding agent reads that an API timeout is 30 seconds, then later the documentation changes it to 5.

October 24, 20263 min readMachine Learning

The real challenge with AI memory isn't how much it can hold, it's knowing what to let go. A recent discussion on long-context models makes this painfully clear: a coding agent reads that an API timeout is 30 seconds, then later encounters updated documentation setting it to 5 seconds. Both facts sit in context, yet the agent clings to the old value. It hasn't forgotten. It simply hasn't updated. This isn't a storage problem; it's a conflict-resolution problem, and it's one the field hasn't fully grappled with.

We spend so much energy celebrating how many tokens a model can ingest that we overlook the more human skill of revision. The same tension appears in how we think about knowledge itself. In our piece on Discover how AI builds meaning from concepts, not just tokens, we explored how models construct understanding from relationships, not raw data. That framing makes the forgetting problem sharper: if a model's knowledge is built from associations, updating one fact risks destabilizing everything tied to it. And when we look at practical applications, like the tagging challenges described in Why AI struggled to tag 126,000 ASMR clips, the cost of stale information becomes tangible. A model that can't forget an outdated label will keep misclassifying content forever.

The architectures being proposed to address this, DeltaNet's prediction-error updates, Dragon Hatchling's Hebbian plasticity, are promising steps, but they raise a deeper question. When should a model overwrite an old memory versus preserve it for context? In human cognition, forgetting isn't failure; it's a feature that allows us to generalize and adapt. AI systems currently lack that discernment. They treat every token as equally authoritative, which means a piece of outdated documentation can silently override newer, correct information. The benchmarks we use don't test for this. Most long-context evaluations ask whether a model can find a needle in a haystack. What we need is a test that asks whether a model can swap the needle when the haystack changes.

The practical consequence is straightforward: until we build memory systems that prioritize recency and relevance over raw recall, AI agents will remain brittle in dynamic environments. Any application that relies on live data, financial analysis, code generation, medical research, will suffer from the same failure mode. The next big advance in AI won't come from longer context windows. It will come from building models that know what to forget.

From Machine Learning

Long-context models can now handle hundreds of thousands, even millions of tokens. But I've been wondering if we're focusing too much on recall and not enough on what happens when information changes.
Say a coding agent reads that an API timeout is 30 seconds. Later, the documentation changes it to 5 seconds. Both versions are still in context, but the agent keeps using 30.

Read the original at Machine Learning

Briefing

Source: Machine Learning
Published: October 24, 2026

Mapping AI's Memory: Where RNNs, Transformers, and SSMs Store What They Know
Someone who has always loved looking at the space between different AI techniques, this time I went a little deeper into the memory trade-offs between RNNs, Transformers and SSMs. I found it interesting because once you start looking at these architectures through the lens of working memory, a lot of the differences become easier to understand. Where does the memory actually live? Is it a compact recurrent state, a growing KV cache, or something closer to the network itself? RNNs keep memory in a recurrent hidden state, which is pretty elegant because the state carries forward step by step. But there is also a bottleneck here as you see, a model can have roughly O(N²) parameters while carrying only roughly O(N) state across time. This means whether RNNs were really doomed because recurrence was a bad idea, or whether the problem was more about the ratio between memory and compute. Transformers make almost the opposite trade-off. During cached inference, instead of compressing the past into one hidden state, they store past representations as key-value entries and attend over them. I think of these almost like little post-it notes: every token leaves behind a key for finding it and a value for what should be remembered. That's extremely powerful, but it also has an interesting property: with weights frozen during inference, the model is managing context rather than turning that experience into durable model knowledge. You get this split between the fixed weights on one side and the fast-changing KV cache memory on the other. Now we have SSMs that bring us back toward fixed-size recurrent memory, but with very different state structures and update rules. Selective SSMs such as Mamba make retention input-dependent, so what gets kept or forgotten depends on the incoming token. Their states don't have to be as small as classical RNN states, but they still compress history into finite memory. And this brings another question does the state have to live in a compressed working dimension, or could it live somewhere closer to the model's internal neuron/connectivity structure? BDH (Dragon Hatchling) is one example I saw and laid a stone that I started with this comparison. It combines linear attention in a high-dimensional neuron space with a low-rank GPU implementation. Its recurrent attention state is an N × D matrix, with N≫D, rather than a materialized N × N connectivity matrix. In the graph interpretation, correlated neuron activity produces Hebbian-like updates to connections, giving working memory a kind of synaptic interpretation. Here working memory and learned connectivity become more closely aligned in structure. But if you think that doesn't mean experience is actually being consolidated into trained weights, and fixed-size state still has a finite information capacity. So I'm definitely not claiming this kills Transformers (or does it ) or solves continual learning. I'm more interested in whether "where does memory live?" is actually a better question than the usual architecture horse race. Are SSMs and these more network-centric architectures actually giving us a better way to handle memory, or are we still running into the same fundamental problem of having to compress history into a finite state? Thoughts? submitted by /u/Pretty_Upstairs9035 [link] [comments]

Topics

LLM memory long-context models memory architectures information update contradictory evidence