binary trees

Sparse attention model reads 94% fewer keys while keeping accuracy

Reading just 30 keys per query instead of 512 is a compelling proof point. ALHR's adaptive routing keeps top-1 accuracy within a few points of a dense model while compressing KV cache by 35x and shifting memory scaling from quadratic to linear. The reliance on a dense teacher during phase one is a practical tradeoff, not a flaw. Full-scale tests are pending, but this direction deserves attention. For more on reducing redundancy in attention, see our piece on two new attention models.