How to make any Sparse Attention / KV Compression look good? [D] [R]
Our take
The recent post by p_nawrot on X, detailing observations on the presentation of research in efficient attention and KV Cache Compression, serves as a bracingly honest critique of a trend within the AI research community. It’s a perspective that resonates deeply with anyone grappling with the complexities of evaluating large language models and their associated optimizations. The core message – that it’s surprisingly easy to make results *appear* impressive even when the underlying improvements are marginal or selectively presented – is a vital one, particularly as the field rushes to demonstrate progress. This echoes concerns raised in our own explorations of Retrieval-Augmented Generation (RAG), specifically around ensuring reliable memory and understanding, as highlighted in Designing a Persistent Knowledge Layer That Refuses to Guess. The ease with which benchmarks can be manipulated, as p_nawrot points out, underscores the need for more rigorous and transparent evaluation methodologies.
The author’s breakdown of common tactics – cherry-picking datasets, isolating contributions to inflate impact, and aggregating metrics to obscure weaknesses – is both cynical and insightful. It’s a reminder that reporting results is as much a craft as the research itself. The discussion of prompt engineering, in particular, highlights a subtle but powerful lever for influencing model behavior and presenting it as a compression achievement. This manipulation is further compounded by leveraging large language models themselves for optimization, potentially masking underlying inefficiencies. The comparison to the use of older, less optimized baselines, then subsequently tuning a method against them, is a particularly damning indictment of practices that prioritize publication over genuine advancement. Relatedly, the ongoing debate around the reliability of automated reviewer tools – and how their feedback might differ from human reviewers – as discussed in For the people who got reviews back from neurips, cvpr, eccv, etc and also tested their paper through an agentic reviewer like the stanford one, how different were the reviews? – adds another layer of complexity to the evaluation landscape.
The identification of "saturated tasks" – benchmarks where even baseline models achieve high scores – is a crucial point. Evaluating improvements on such tasks provides a false sense of progress and can obscure the true limitations of a technique. The observation that larger models might simply be absorbing compression through increased capacity, rather than demonstrating genuine efficiency gains, challenges the conventional narrative of progress. This is especially relevant given the focus on efficient workflows within RAG architectures, where optimizing both retrieval and generation is paramount, as explored in RAG Workflow and Loop Engineering: The Dispatcher That Decides When to Loop and When to Stop. The emphasis on carefully chosen metrics and selective reporting risks creating a distorted picture of the field's actual progress.
Ultimately, p_nawrot's post isn’t a condemnation of the researchers involved, but a call for greater transparency and rigor in evaluating AI-native technologies. It’s a reminder that the pursuit of efficiency and innovation should not come at the expense of honest reporting and a critical assessment of limitations. The increasing complexity of these models and their evaluation demands a shift towards more nuanced and comprehensive benchmarks, and a greater willingness to acknowledge areas where progress remains elusive. The question now becomes: how can the community collectively develop evaluation practices that discourage superficial optimizations and incentivize genuinely transformative advancements in AI efficiency?
Original Article - https://x.com/p_nawrot/status/2089315591010079034
I've spent the last few years working on efficient attention and KV Cache Compression. I've read many papers, dug deep into reference or official implementations of methods, and inspected appendices—and I think I've learned a few things. One of them is definitely "how to make things look good, even when they aren't."
I'm guilty too, but trying to get better every day.
1. For single-hop retrieval, make sure there are no distractors and context is useless
The three most cooperative settings for compression / sparsity are:
- Needle in a haystack with a single OOD key-value pair and context built out of a repeated sentence or irrelevant background text.
- Contaminated benchmarks from years ago for which models don't even look at the context anymore.
- Few-shot in-context learning, where extra shots are useless and don't improve the accuracy over 0-shot.
With 1) synthetic tasks, 2) real-data QA, and 3) in-context learning, you get a semblance of broad coverage without the inconvenience of testing much diversity within any of them. Most tasks in these settings should pass under Sliding Window Attention, so it doesn't matter that much whether your method works. Combine it with SWA and you should be good to report 5–10x compression or sparsity.
2. NEVER isolate your contribution
Short context: Most of a dense model's performance is recovered by a local window + attention sinks + the ability to retrieve an answer sentence that is largely n-gram matchable with the question. The remaining part is significantly more difficult, but it's neither relevant to nor the subject of this post.
- Say prior work developed an algorithm X, and its implementation separately keeps a local window of 256 tokens. You find that your method is on par with X in a matched setting, but better and more stable with a window size of 512—let's go, don't look back.
- Do the same with block size. Smaller blocks can give you finer granularity and more precision in retrieval, so keep their old block size and make yours smaller. Ignore the fact that things may get slower due to irregular memory accesses, etc. Those were historical decisions; respect them. 🤡 Write: “We used the authors’ recommended hyperparameters.”, then spend weeks tuning your method.
- The same trick works for speed. LLMs are pretty good at writing Triton now. Keep the baseline algos exactly as they were written in 2023, then ask an LLM for a custom Triton kernel for yours. Extra cleverness if, by using a more efficient implementation, you can hide that your method does more work. You're just optimising your method, no?
- Prompts are the cherry on top. Move the question before the context so the model knows what to filter out, then present the result as lossless compression. Never share the prompts after tuning them.
Don't tune the baselines to reject your paper; tune yours until it's accepted.
3. Use aggregated metrics to hide areas where your method doesn't work
RULER has 13 tasks:
- 6 NIAH tasks satisfy the first point.
- 2 QA tasks use datasets from years ago.
- VT also has a lot of irrelevant context.
To be clear: This isn't a critique of RULER; imo it's still incredibly useful. It's just an example of potential improper use.
Report only the aggregate; maybe, in the limitations section at the end, briefly mention that your method degrades on the NIAH-MK3, which actually stress-tests lossless compression.
4. Enjoy saturated tasks
Imagine evaluating on two tasks:
- The most recent math exam / olympiad from a week ago, which isn't yet in the training data.
- A benchmark on which a recent family of open models—1B, 10B, and 100B—all scored 80%.
On the former task, before compression gets a chance to do any damage, the 1B and 10B models already score 0%; the 100B model starts at 50%, and its performance drops monotonically as compression increases. On the latter, all model sizes tolerate substantial compression, and the 100B model tolerates more than the 1B and 10B models.
Don't ask whether the larger model is simply using its extra parameters and hidden-state capacity to absorb compression in a setting where those resources aren't needed to solve harder questions. That definitely isn't what's happening.
Extras
- AIME has 30 samples. You did 4 seeds. Your method scores 80, and the baseline scores 79—bold your 80 and say that it surpasses the baseline. Statistics doesn't exist. Bonus points for your efficiency method surpassing the baseline and setting a new SOTA. 🤡🤡
- Pick a baseline, optimise it with your method, and plot a beautiful quality–efficiency curve against the original implementation. Then stop. Don't ask whether a simpler route—a smaller dense model, KV-cache quantisation or offloading, or a better system configuration—reaches a better operating point. Improving your baseline is basically the same as improving the frontier.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience