ICML Position Track: Want Better ML Reviews? Stop Asking Nicely and Start Incentivizing with a Credit System [D]

The experience of submitting and reviewing papers at machine learning conferences is, let’s face it, often fraught with frustration. The sentiment expressed in this ICML position paper—that perhaps the real AGI was the connections made alongside the unpleasantries of the review process—resonates deeply. As highlighted in this piece, a lack of accountability and incentives within the current system allows undesirable behaviors to flourish, hindering constructive discussion and ultimately impacting the quality of research disseminated. The author’s proposal for a credit system, designed to reward positive contributions and introduce accountability, offers a compelling and much-needed intervention. It’s a recognition that relying on goodwill and polite guidelines simply isn't enough to address systemic issues. Considering the challenges explored in "What if a model could only learn what trusted LoRA adapters can express? [R]," which touches on the vulnerabilities of fine-tuning and data integrity, ensuring a robust and reliable review process becomes even more vital for maintaining the credibility of the field. Similarly, the upcoming AMA with Raffi Krikorian (CTO, Mozilla) — AMA on the State of Open Source AI [D] demonstrates the growing importance of community collaboration and shared responsibility in the AI landscape, themes directly relevant to improving conference workflows.

The credit system’s potential extends far beyond simply rewarding reviewers. The inclusion of refundable submission fees, dependent on the quality of the submission as assessed by reviewers, is a particularly intriguing prospect. This mechanism could disincentivize low-quality submissions and encourage authors to invest more effort in their work. Mobilizing non-author reviewers—removing the inherent conflict of interest present when authors also serve as reviewers—is another brilliant suggestion, tapping into a valuable pool of expertise often overlooked. The author is rightly cautious, acknowledging the system is “far from perfect,” but its core concept speaks to a fundamental truth: incentivizing positive behavior is far more effective than simply discouraging negative behavior. Current conference structures often treat the review process as an obligation, rather than an opportunity for valuable contribution and intellectual exchange. This shift in perspective, facilitated by a system of rewards and accountability, has the potential to significantly elevate the quality of research and the overall conference experience.

The fact that this proposal is being presented within ICML’s position paper track—a platform designed for exploring novel ideas—is itself a positive sign. It underscores a growing willingness within the community to critically examine and improve the foundations of machine learning research. While the implementation details undoubtedly require careful consideration and iterative refinement, the underlying principle of leveraging incentives to foster a more collaborative and rigorous review process is sound. The current reliance on a largely volunteer-based system, with limited oversight and accountability, has demonstrably failed to prevent issues like superficial reviews and delayed feedback. The proposed credit system offers a pragmatic and scalable solution to these challenges, promising a more efficient and equitable process for both authors and reviewers. The release of “MIRA: Multiplayer Interactive World Models trained on Rocket League [R]” highlights the power of collaborative environments and shared knowledge, principles that could be directly applied to improve the conference review ecosystem.

Looking ahead, the success of any such system will hinge on thoughtful design and widespread adoption. How will points be allocated fairly and consistently? How can we prevent gaming the system? And perhaps most importantly, how can we ensure that the incentives align with genuine quality and rigor, rather than superficial metrics? The conversation sparked by this position paper is a crucial first step. It’s a call to action for conference organizers, reviewers, and authors alike to actively participate in shaping the future of machine learning research—a future where the process of sharing and validating knowledge is as rewarding as the discoveries themselves. Will we see a widespread adoption of incentive-based review systems across major AI conferences in the next five years, fundamentally altering the way research is evaluated and disseminated?