Accuracy preservation is the wrong frame for this discussion. When NVIDIA's Ling-3.0-flash drops from 96.89% to 96.82% on GSM8K across 1,319 test questions, the real story isn't whether that 0.07-point gap is "quality-neutral", it's whether the optimization bundle is trustworthy enough to ship into real workflows. We think the community should demand more evidence before accepting the claim, and the evidence path is clearer than most realize.
The aggregate score hides the signal. Two configurations that agree on 1,300 questions and disagree on 19 could look identical in summary, or one could systematically fail on the hardest reasoning steps while scoring higher on routine arithmetic. The paired per-question view is the only way to distinguish those cases. Without it, an average is just a number. This is exactly the kind of scrutiny that matters when you consider how Perplexity brings its AI agent home, running locally on Nvidia hardware you own, a local deployment means users rely on the model's consistency without a cloud fallback. If the optimized configuration trades one type of error for another, the user sees a regression even if the aggregate looks fine.
The real challenge is evaluation budget. Three changes shipped together: a new MoE backend, an FP8 output head, and NEXTN speculative decoding. A paired per-question analysis across multiple random seeds would isolate which component introduces the variance. Repeated seeds matter because one run could land on a lucky or unlucky sequence of tokens, especially with speculative decoding in the loop. A declared equivalence margin, say, 0.5% on GSM8K, would turn the claim from a statement of fact into a testable hypothesis. Without that margin, "no meaningful quality regression" is an opinion dressed as data.
What we would recommend is a two-step check. First, publish the per-question agreement matrix for the 1,319 items, color-coded by which configuration won and by what margin. Second, run the same comparison on a second task set, something like MATH or MMLU, where the precision change and decoding optimization might behave differently. If the error trade-offs look similar across tasks, the bundle is safer. If they diverge, the evaluation budget should expand before any deployment decision. The takeaway is straightforward: aggregate accuracy alone is insufficient for a claim of quality neutrality, and the burden of proof should include per-question transparency, repeated seeds, and a declared margin. Until then, treat the 96.82% as a promising but unverified number.