Reviewing 4 papers for AAAI 2027 and none have code, Reject? [D]

The recent post on Reddit, detailing a reviewer's experience with AAAI 2027 submissions lacking code and data, highlights a growing tension within the AI research community. It’s a familiar frustration – empirical claims presented without the means to verify them. The reviewer’s measured response – flagging the issue and requesting anonymized code in the rebuttal – demonstrates a pragmatic approach, acknowledging the practical constraints of review while upholding the principles of reproducibility. This situation echoes the broader challenges we face in ensuring the rigor and reliability of AI research, a topic we’ve explored previously in articles like Mastering the AI Project Cycle: From Concept to Production, emphasizing that robust AI systems require more than just model selection and data feeding; they demand a complete and verifiable project lifecycle. The reviewer’s concern isn’t an isolated incident, and the increasing emphasis on open science and reproducible research makes this lack of supporting materials increasingly problematic.

The argument presented within the thread – that reviewers rarely have time to audit code – is a sobering reality. While the AAAI’s own guidelines mandate code/data submission, the practical limitations of the review process often prevent thorough verification. This raises a critical question: how can we balance the ideal of full reproducibility with the realities of reviewer workload and the legitimate concerns of authors regarding intellectual property and funding restrictions? The post’s observation that a paper’s credibility is severely undermined by unverifiable empirical results is entirely valid. It speaks to a deeper issue – the increasing reliance on benchmarks and metrics in AI research, which can incentivize superficial results and discourage exploration of less easily quantifiable approaches. Our own work on Millwright — experimenting with an end-to-end machine learning framework in Rust underscores the complexity of building reproducible ML systems, and the benefits of open-source collaboration in fostering transparency and validation.

The crux of the matter lies in finding a sustainable middle ground. A blanket “auto-reject” for missing code is likely overly punitive, particularly given the often-valid reasons authors may have for delaying release. However, the current situation, where empirical claims are presented without any possibility of verification, is simply unacceptable. We need to move beyond the superficial checklist compliance and foster a culture of genuine transparency and accountability. Perhaps a tiered system could be implemented, where papers with substantial empirical components are required to provide at least minimal code snippets or data samples, with options for anonymization or delayed release. The reviewer's proactive approach – explicitly flagging the issue and requesting code in the rebuttal – sets a positive example, but a more systemic solution is needed to ensure the integrity of the research process.

Ultimately, the conversation sparked by this Reddit post serves as a vital reminder that the future of AI research hinges on its ability to demonstrate not just what *can* be achieved, but *how* it was achieved. The pressure to publish quickly and demonstrate impressive results has, at times, overshadowed the importance of rigorous methodology and reproducible findings. Moving forward, it’s crucial to prioritize transparency and accessibility, ensuring that the foundations of AI innovation are built on a bedrock of verifiable evidence. What mechanisms, beyond the current guidelines, can conferences and journals implement to proactively encourage – and even require – a higher degree of reproducibility without unduly burdening authors or reviewers?