greedy decoding
greedy decoding at Beyond Market Intelligence is a file of 2 stories. The newest of them: “Building a GPT-2 KV Cache to Measure Real AI Performance Gains” and “A small language model that teaches itself column arithmetic.”. Building a GPT-2 KV cache in PyTorch as a learning project is exactly the kind of hands-on work that reveals where the math meets reality. This is the kind of result that makes you question what small models actually need. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work… The list below is every greedy decoding story on Beyond Market Intelligence, newest first.
Building a GPT-2 KV Cache to Measure Real AI Performance Gains
Building a GPT-2 KV cache in PyTorch as a learning project is exactly the kind of hands-on work that reveals where the math meets reality. The numbers tell a fascinating story. At context length 1,000, the FLOP estimate suggests a 1,000x reduction with caching, yet actual per-token latency only drops about 6x. That gap is worth exploring. The memory measurements are impressively clean, with requested bytes matching the formula to zero after accounting for a constant cuBLAS workspace offset.
A small language model that teaches itself column arithmetic.
This is the kind of result that makes you question what small models actually need. A 348M-parameter model, trained on 22.7B tokens, hits 99.4% across nine arithmetic sub-tasks, beating GPT-3's 175B few-shot numbers by a wide margin. The clever part isn't just the accuracy; it's that the model learned place values it was never taught, inventing "millions" and "ten-millions" from zero examples. That's not brute force. That's pattern recognition. We've covered how AI reshapes data ownership, and this feels like a natural extension