Paper lengths, and reasonable assumptions in ML conferences. [D]

The frustrations voiced in this recent post regarding paper reviews at major machine learning conferences resonate deeply within the theoretical AI community. The author's observation that increasingly, rejections stem from subjective criticisms – “The math is difficult to understand,” or “Certain terminology is not explained” – rather than substantive flaws in the underlying theory, highlights a growing disconnect between the demands placed on reviewers and the constraints imposed by conference formatting. This echoes concerns raised in discussions surrounding [Link plots/figures in NeurIPS rebuttal [R]], where authors struggle to effectively communicate complex results within the rigid structures of conference proceedings, and is further complicated by the evolving landscape of AI-assisted review, as explored in [NeurIPS 2026 AI-generated reviews [D]]. The author’s point about the inherent escalation of prerequisite knowledge required for theoretical papers is particularly astute; it's a natural consequence of pushing the boundaries of understanding, and shouldn’t be unfairly penalized.

The core issue, as the author aptly puts it, is a dissonance between the "unlimited appendices" policy, designed to mitigate reviewer fatigue, and the expectation that papers should be entirely self-contained. While the intention behind the appendices is laudable – allowing authors to include supplementary material without inflating the core paper’s length – the rule explicitly states reviewers aren't *expected* to read them. This creates a paradoxical situation where authors are penalized for presenting complex material that may necessitate deeper exploration, even though the system ostensibly allows for it. The shift from constructive criticism – "This has been done before" or "Why don't you compare with X, Y, Z" – to vague pronouncements about difficulty underscores a potential decline in reviewer engagement and a reluctance to acknowledge the inherent challenges of evaluating highly theoretical work. The author's comparison to teaching real analysis is spot on; some concepts are simply difficult, and expecting a reviewer to immediately grasp them without dedicated study is unrealistic.

The broader implications of this trend are concerning. If theoretical papers are consistently rejected on grounds of perceived difficulty, it could stifle innovation and discourage researchers from pursuing challenging, groundbreaking work. The current system, inadvertently, seems to favor papers with more accessible, empirical findings, potentially at the expense of fundamental theoretical advancements. This isn't about demanding reviewers become experts in every subfield; it’s about fostering a culture of intellectual honesty and encouraging reviewers to acknowledge when a paper’s complexity lies beyond their immediate expertise. The suggestion of a clarifying rule – “Don't be a dick. If you don't have the pre-requisite knowledge, say so, review what you can” – while blunt, encapsulates the essence of the problem: a need for greater transparency and humility in the review process. This is also a discussion that ties into how we evaluate research, as explored further in [How exactly does the NeurIPS meta reviewer response work? [D]].

Looking ahead, conference organizers need to seriously consider how to better support the evaluation of theoretical submissions. Perhaps a tiered review system, where papers are initially assessed by reviewers with relevant expertise, could mitigate the issue of superficial rejections. Alternatively, a more explicit acknowledgement of the appendix’s potential value, coupled with guidelines for reviewers on how to engage with supplementary material, could foster a more nuanced and productive evaluation process. The question remains: how can we create a system that rewards rigorous theoretical work while ensuring fairness and intellectual honesty in the peer review process, especially as AI tools increasingly influence that process?