Post-Transformer architectures

Exploring the Next Wave of AI Architectures Beyond Transformers

The race to move beyond the Transformer is producing a fascinating zoo of architectures, from Mamba's selective state spaces to Dragon Hatchling's emergent attention.

4 min readMachine Learning

The most interesting thing about the post-Transformer landscape isn't any single architecture, it's the quiet consensus forming across labs that pure scale has hit its practical ceiling. When you line up Mamba-2, RWKV-7, Titans, and the hybrid hybrids from Alibaba, Moonshot, and NVIDIA, the pattern is unmistakable: everyone is hunting for more intelligence per watt, per dollar, per token. That shift matters more than any specific state-space model, because it signals that the industry's next competitive edge won't come from throwing more GPUs at a problem. It will come from designing systems that reason harder with less.

For teams still building on standard transformer stacks, this is the moment to start paying attention to how memory is handled. The comparison table in the original breakdown highlights something practical: test-time memory is becoming a first-class design choice, not an afterthought. Titans uses a neural long-term memory module updated by "surprise" during inference. RWKV-7 keeps constant memory with no KV cache. Dragon Hatchling leans on Hebbian-style synaptic updates. These aren't academic curiosities, they're direct answers to the context-window bloat and inference costs that plague production deployments today. If your roadmap involves long documents, agentic workflows, or real-time reasoning, the architecture you choose in 2026 will determine whether your latency and energy bills stay sane.

There's also a deeper lesson here about how breakthroughs propagate. A lot of major shifts in AI start as research papers and slowly become the backbone of industry tools. We've seen that pattern play out with formal reasoning methods, where uncovering system flaws before they exist moved from academic niche to practical engineering discipline. The same trajectory is visible in these hybrid architectures: what looks like a fragmented set of experiments today will likely consolidate into a few dominant patterns tomorrow. The winners won't be the labs with the most exotic mechanism, but those that make their approach easy to adopt, tune, and deploy on existing hardware.

What we're watching is not a revolution in the sense of a single breakthrough moment. It's a gradual, deliberate re-engineering of the transformer's weakest points, attention's quadratic cost, static weights, and poor test-time adaptation. The fact that so many approaches overlap, linear attention plus softmax layers, recurrent cores with attention blocks, memory modules with delta rules, suggests the field is converging on a hybrid toolkit rather than a clean replacement. That's good news for practitioners: you won't need to bet everything on one unproven paradigm. You'll be able to mix and match components based on your workload.

The specific detail to watch is the ratio of linear-attention to softmax-attention layers in models like Qwen3-Next and Kimi Linear, both at 3:1. That's not a random number. It's a practical compromise that preserves reasoning quality while cutting memory overhead. If those ratios keep shifting toward linear dominance without accuracy loss, that's the signal that the post-transformer era is truly here. Until then, treat every architecture announcement with healthy skepticism, but keep your eyes on the memory budget. That's where the next real gains will be won.

From Machine Learning

Went down the rabbit hole to map the different Post-Transformer architectures. Various directions include Dragon Hatchling (BDH), Mamba, RWKV, xLSTM, Titans, and a bunch of hybrid models coming from different AI Labs. My interest areas are architectural changes that can improve reasoning, memory, and adaptation within practical compute and energy budgets.

IMO, the next frontier might be less about scaling alone and more about getting more intelligence out of limited resources. That's what got me looking into these architectures and putting together this comparison table.

Read the original at Machine Learning