Scientific rigor depends on reproducibility, yet the peer review process has long treated code and compute as optional extras rather than core evidence. A prospective reviewer recently asked a practical question: how do you decide whether to allocate your own compute resources to validate a paper's results, especially when the accompanying code is often difficult to run? This is not a hypothetical concern. As the line between data analysis and data storytelling continues to blur, the reviewer's willingness to press "run" may be the only real check on a paper's claims. We believe that compute-backed validation should become a standard expectation, not a heroic act, and that the burden must shift from individual reviewers to the systems that publish the work.
The tension here mirrors what we have explored in How AI Is Reshaping the Peer Review Process for Better or Worse. AI tools are already generating reviews and writing sections of papers, which raises a hard question: if machines can produce plausible text without understanding the data, how do we verify that the numbers behind the text are real? The answer cannot be "trust the authors." Running the code yourself is the most direct verification method, but it is rarely straightforward. The reviewer in our story notes that the code in papers is not usually easy to run. This is a systemic failure. When a paper's repository lacks environment specifications, dependencies, or even a README, the reviewer must either spend hours debugging or simply assume the results hold. Neither option serves science.
What would change if journals and conferences required a reproducible execution environment as a condition of submission? We are not talking about cloud credits for every reviewer, but about a baseline: a containerized workflow that a reasonably skilled practitioner can launch without reverse engineering. The compute cost for a single validation run is often trivial compared to the cost of publishing a false result. This is where the conversation about efficiency meets responsibility. In Explore smarter AI models to shrink your data carbon footprint, we discussed how intentional compute allocation can reduce waste. The same logic applies here. A small, reproducible test harness uses less energy than a reviewer manually recreating the analysis from scratch, and it produces a verifiable artifact that future readers can inspect.
The practical takeaway for anyone preparing to review is straightforward: before you accept a review assignment, check whether the authors have provided runnable code and clear instructions. If they have not, consider asking the editor to make reproducibility a condition of the review. You are not being difficult; you are protecting the integrity of the process. The reviewer who asked this question is right to wonder how to allocate their compute. The real answer is that the allocation decision should rarely be theirs alone. It should be built into the submission pipeline, enforced by standards, and supported by the community. The next time you evaluate a paper, ask yourself whether you could reproduce its central figure in an afternoon. If the answer is no, the problem is not with your compute budget. It is with the system that asked you to judge results you cannot verify.