The current approach to benchmarking is failing us, and it is time to admit it. When frontier labs burn through tens of thousands of prompts just to report marginal gains, they are not building confidence; they are burning carbon for a veneer of rigor. The Gemini example, with its 30,000 prompts, is a case in point. Run the same suite again after each iteration, and you are simply multiplying the cost without adding meaningful insight. This is not a niche concern. It is a systemic habit that other organizations will copy, and the waste will compound across the industry.
What does this mean for you, the practitioner? It means every time you evaluate a model, you are likely paying for more than compute. You are paying for a methodology that prioritizes volume over signal. The standard pass@k metric does not tell you if Iteration A is genuinely better than Iteration B. It tells you how many tries the model needed to get lucky. That is not confidence. That is a coin flip dressed in a benchmark report. If you are building agents or fine-tuning models, you need to know early whether you are on the right trajectory, not after spending thousands of dollars on a harness that tells you what you already suspected.
There is a better way, and it is not hypothetical. The bayesbench package, built by the author of this piece, applies Bayesian techniques to model evaluation. The idea is simple: model the confidence of your results directly, so you need fewer samples to reach a decision. If Iteration A is meaningfully better than Iteration B, you will know it faster and with less compute. If the models are too similar to differentiate, the method will tell you that too, saving you from chasing noise. This is not about ditching benchmarks entirely. It is about making them smarter and more honest about what they can and cannot tell you.
The practical takeaway is this: stop treating benchmark suites as a default ritual. Start asking whether your evaluation method is actually reducing uncertainty or just consuming resources. If you are evaluating agents, this matters even more, because the cost of data collection and inference scales quickly. The demo on Hugging Face lets you test these ideas today. The package is open for contribution. The question is not whether evaluation will shift; it will. The question is whether you will be part of building that shift or left paying the carbon bill for a confidence you never really had.