Bad but typical NeurIPS experience? [D]

The recent Reddit post detailing a deeply frustrating NeurIPS review experience – one marked by adversarial reviews, unresponsive reviewers, and a seemingly arbitrary scoring system – isn't surprising, but it’s a stark reminder of the systemic challenges plaguing top-tier AI conferences. This individual’s account echoes concerns we've previously explored, such as the need for more rigorous standards for reproducibility, as highlighted in [It's time to desk reject papers that don't include code that can reproduce the results [D]], and the persistent issue of reviewers failing to adjust scores after satisfactory rebuttals, a topic we addressed in [NeurIPS 2026: If the rebuttal addresses your concern, please raise your score [D]]. The lottery-like nature of these peer-review processes, where a researcher's fate can hinge on the whims of a few individuals, creates a climate of anxiety and undermines the pursuit of genuinely impactful research. It’s a system that rewards luck as much as, if not more than, merit, and it disproportionately affects those who challenge the status quo or whose work doesn't neatly fit into pre-existing paradigms.

The adversarial review phenomenon, where reviewers seem actively intent on rejecting a paper regardless of its merits, is particularly troubling. While rigorous critique is essential for scientific progress, outright antagonism serves only to discourage innovation and create a hostile environment for researchers. The lack of responsiveness from reviewers and the area chair (AC) further exacerbates the problem, leaving authors feeling powerless and unheard. The reviewer's comment about addressing concerns but maintaining a reject score highlights a worrying disconnect between constructive feedback and objective evaluation. It suggests a potential bias or unwillingness to reconsider initial judgments, even in the face of compelling evidence to the contrary. This situation isn't unique to NeurIPS; similar issues have been reported across various fields, indicating a broader problem within academic peer review. The rise of AI-powered research, as explored in [TechCrunch Mobility: Two roads diverged — for robotaxis], further complicates this landscape, as novel methodologies and unexpected results may face greater scrutiny from reviewers unfamiliar with the underlying techniques.

The consequences of this toxic system extend beyond individual researchers. It stifles creativity, discourages risk-taking, and ultimately hinders the advancement of the field. When researchers are preoccupied with navigating the complexities of the review process rather than focusing on groundbreaking discoveries, the entire community suffers. Addressing these issues requires a multi-faceted approach. Conference organizers should implement stricter reviewer guidelines, emphasize the importance of constructive feedback, and explore mechanisms for identifying and mitigating adversarial reviewing behaviors. Automated tools, while not a panacea, could potentially assist in identifying biased or unhelpful reviews. More transparency in the review process, such as making reviewer comments publicly available (while protecting reviewer anonymity), could also promote accountability and encourage more thoughtful evaluations. Ultimately, a shift in culture is needed, one that prioritizes rigorous, fair, and respectful peer review over arbitrary scoring and personal biases.

Looking ahead, the increasing reliance on AI in research—both in the creation and evaluation of papers—presents both opportunities and challenges. While AI could potentially be used to automate aspects of the review process and detect biases, it also raises concerns about algorithmic fairness and the potential for reinforcing existing inequalities. The question remains: how can we leverage AI to improve the peer-review process while safeguarding against its potential pitfalls and ensuring that the pursuit of knowledge remains a collaborative and equitable endeavor? It’s a critical question that will shape the future of AI research and the conferences that showcase it.