context length
6 stories filed under context length on Beyond Market Intelligence. The newest of them: “Building a GPT-2 KV Cache to Measure Real AI Performance Gains”, “Linear Recurrence Matches Transformer Accuracy at a Fraction of the Cost”, and “Two paths to a smarter decision model: equal accuracy, unequal cost”. Building a GPT-2 KV cache in PyTorch as a learning project is exactly the kind of hands-on work that reveals where the math meets reality. When you budget-match a linear recurrent model and a transformer down to 15 million parameters on a long-context retrieval task, something unexpected happens: neither learns much. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work… The list below is every context length story on Beyond Market Intelligence, newest first.
Building a GPT-2 KV Cache to Measure Real AI Performance Gains
Building a GPT-2 KV cache in PyTorch as a learning project is exactly the kind of hands-on work that reveals where the math meets reality. The numbers tell a fascinating story. At context length 1,000, the FLOP estimate suggests a 1,000x reduction with caching, yet actual per-token latency only drops about 6x. That gap is worth exploring. The memory measurements are impressively clean, with requested bytes matching the formula to zero after accounting for a constant cuBLAS workspace offset.
Linear Recurrence Matches Transformer Accuracy at a Fraction of the Cost
When you budget-match a linear recurrent model and a transformer down to 15 million parameters on a long-context retrieval task, something unexpected happens: neither learns much. Accuracy hovers near chance for both. The real story is cost. At 8,192 tokens the linear model runs 2.37× faster, and training time opens a 6-7× gap that only widens with scale. This negative result from Jev's decision-lab is exactly the point, small decision models are accuracy-starved regardless of substrate, so the winning choice becomes efficiency.
Two paths to a smarter decision model: equal accuracy, unequal cost
Two paths to the same decision model, matched within 0.02% on parameters, yet the cost difference is stark. A linear recurrent backbone and a standard attention transformer were built on identical budgets for a long-context retrieval task. At 8K tokens, the linear model runs 2.37× faster. Training? Six epochs took 30 minutes versus 40-45 minutes *per epoch* for the transformer. Neither substrate learned much at 15M parameters, accuracy hovered near chance. The real insight is where cost lives.
Nvidia's simple math cuts costly recomputation across AI models
Swapping models mid-session has always meant paying the full prefill tax again, until now. Nvidia's researchers found that a simple linear mapping can transfer a KV cache between compatible models, bypassing the expensive recomputation that bogs down multi-LLM agentic workflows. The math is refreshingly direct, not a heavyweight neural network. On tested pairs, this approach runs up to 25 times faster while keeping nearly all accuracy. It's a practical fix for a costly bottleneck, and it points toward leaner long-horizon AI systems.
Scaling DNA modeling with linear attention that remembers what matters
A 25% recall score on a four-token DNA vocabulary isn't just a stumble; it's a sign that the compressed state in linear attention is dropping the thread. This isn't a niche bug, as HyenaDNA hits the same wall. The core tension is clear: state compression trades memory for recall, and long contexts expose that trade-off brutally. We're watching a fundamental limit being tested, not a simple fix.
Small open models match top AI on Swedish medical exams
A 4-billion-parameter model just hit 87% on Swedish medical licensing questions, matching what o3 scored months ago. That's worth pausing over. The gap between open-weight models and frontier systems is closing faster than most expected. Qwen3.5-4B, with reasoning enabled and no post-training, nearly matches a model that was state of the art in 2025. The early exit trick from the S-GRPO paper is a smart fix for those looping reasoning traces, and it works.