Long Context
Long Context on Beyond Market Intelligence: a running collection of 2 stories we have gathered and hand-picked because they are worth your time. Every post here touches on long context in some way — the news, the analysis, the deep dives, and the occasional surprise find. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work with data. New stories are added to this page as we find them, so check back if you want to keep up with what is happening around long context, or subscribe to the RSS feed to get them as soon as they are published. Browse the collection below, or head back to the homepage to see everything Beyond Market Intelligence is covering right now.
I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB [R]
A remarkable achievement in efficient AI development has emerged: a 250M parameter language model, SHADOW-250M, deployed in a remarkably compact 60 MB footprint. Trained on 30B tokens and quantized to under 2 bits, this model achieves 400 tokens/second on a standard laptop CPU – no GPU required. Notably, it leverages a unique long-context system compressing older tokens to disk for retrieval, enabling up to 100 million tokens of history.
![Seeking collaborators for scaling and independent evaluation of a new recurrent language model architecture (preprint + code) [R]](https://preview.redd.it/b0u6q9a46ndh1.jpg?width=140&height=98&auto=webp&s=dbb02d2e0fc85305a04e37864167c2578891d46c)
Seeking collaborators for scaling and independent evaluation of a new recurrent language model architecture (preprint + code) [R]
Researchers have introduced DABSN (Dynamic Adaptive Bias State Network), a novel recurrent language model architecture demonstrating promising results in reasoning, memory, and long-sequence tasks. The initial preprint and accompanying code—available in PyTorch, C++, and Triton—detail the architecture’s behavior and performance across benchmarks like MQAR and A5/60. Early language modeling experiments with a 24M parameter model have yielded unexpectedly strong results, prompting a second paper focused on scaling and long-context behavior. Collaboration is sought for independent reproduction, evaluation design, and access to larger GPU resources.