CoWindow Attention

Two new attention models slash redundancy to stretch context further

Attention research has a habit of making the same trade-off: sacrifice coverage for speed. Two new papers from this author take a different route. CoWindow Attention distributes distant context across KV heads using complementary windows, so each head stays sparse while the collective sees everything. MassAlloc Attention keeps full QK scoring but uses softmax statistics to skip low-value computation. The results at 128K tokens are compelling, 7.4x forward speedups without learned routers. For deeper context on sparse attention architectures, see our coverage of Monodratic.