It's time to desk reject papers that don't include code that can reproduce the results [D]

The recent Reddit post highlighting a concerning trend in machine learning conference paper reviews – the widespread absence of reproducible code – strikes at the very heart of scientific rigor. As review season for NeurIPS wraps up, the reviewer’s observation that only 1 out of 12 papers provided full, runnable code is deeply troubling. This echoes concerns raised in discussions surrounding reviewer scoring, as seen in NeurIPS 2026: If the rebuttal addresses your concern, please raise your score, suggesting a broader need for greater accountability within the peer review process. The fact that even among the papers *providing* code, a significant portion contained demonstrable bugs further underscores the gravity of the situation. This isn't merely an issue of convenience; it’s a fundamental threat to the trustworthiness of published research.

The core problem, as the author aptly points out, lies in the current incentive structure. There’s minimal cost to withholding code during the review process, while the risk of exposing flaws through code release is perceived as potentially leading to rejection. This creates a perverse dynamic where researchers might be disinclined to share their work, hindering the ability of the community to independently verify results and build upon existing findings. The observation is particularly relevant given the rapid proliferation of AI-powered tools and the potential for subtle errors to have cascading effects. Related to this, the community is seeing increasing efforts to self-promote and share projects, as evidenced by D-Self-Promotion Thread, which demonstrates a desire for greater transparency and collaboration, albeit outside the formal peer review process. The reliance on trust and reputation within the ML community is being tested as models become increasingly complex.

The consequences of this lack of reproducibility extend far beyond individual papers. It erodes the collective knowledge base and impedes the progress of the field. While the call for "real penalties" on code withholding is a strong one, implementing such measures presents its own challenges. How do you fairly assess code quality and enforce compliance without unduly burdening researchers? Perhaps a tiered system, where code submission is mandatory for certain categories of papers or those claiming significant advancements, could be a viable starting point. Furthermore, investing in better tooling and infrastructure to facilitate code review and execution would be essential. Automated testing and sandboxed environments could help reviewers identify bugs more efficiently and provide more constructive feedback. The broader implications for the future of AI development, especially concerning safety and reliability, cannot be overstated. We are seeing the impact of these concerns across various sectors, as highlighted in TechCrunch Mobility: Two roads diverged — for robotaxis, where reproducible results and verifiable safety are paramount.

Ultimately, addressing this reproducibility crisis requires a cultural shift within the machine learning community. We need to move beyond the perception of code sharing as a potential liability and embrace it as a fundamental responsibility. The benefits of open and reproducible research – increased trust, accelerated innovation, and improved quality – far outweigh the perceived risks. The question now is not *if* we should mandate code release, but *how* we can do so in a way that is equitable, practical, and truly fosters a more robust and reliable scientific ecosystem. How will the community balance the desire for rapid progress with the imperative of verifiable results?