The peer review process in computer science conferences has become a source of growing frustration for many researchers, particularly when faced with wildly divergent reviewer assessments that seem to shift unpredictably during rebuttal periods. The recent ECCV experience shared by a researcher receiving scores of 1/3, 4/3, and 4/5 highlights a systemic issue that echoes across multiple venues, including [So Confused about Polarizing ICML Reviews [D]](post/so-confused-about-polarizing-icml-reviews-d-cmnw2l1kg0aozzxsx9lnibdjp) and [Seems ICML is rejecting MANY unanimous positively rated papers [D]](post/seems-icml-is-rejecting-many-unanimous-positively-rated-pape-cmom5g7ys00e7jfqbji8hrqi7). These cases reveal how the current system can leave authors questioning not just their work's merit, but the fundamental reliability of academic evaluation itself.
What makes this situation particularly troubling is the implicit contract that reviewers should provide consistent, principled assessments. When a reviewer who initially assigns the lowest possible score suddenly suggests they might upgrade their evaluation after rebuttal, it creates uncertainty that undermines the entire process. This phenomenon speaks to deeper issues within our review culture: the pressure to provide quick assessments under tight deadlines, the lack of clear calibration across reviewers, and the ambiguous role that rebuttals should play in evaluation. Many researchers echo concerns captured in discussions like Many times I feel additional experiments during the rebuttal make my paper worse, where the expectation to continuously expand work during review cycles becomes counterproductive.
The human cost of this uncertainty cannot be overlooked. Authors invest enormous emotional and intellectual energy preparing responses, often working through sleepless nights to address every critique comprehensively. Yet the reward for this dedication remains unclear when reviewer positions appear fluid rather than grounded in consistent standards. This dynamic particularly affects early-career researchers who may lack the institutional knowledge to navigate these ambiguous situations effectively. The stress of potentially having to conduct additional experiments under compressed timelines, only to face the same reviewer who previously deemed the work unworthy, creates a paradox that serves neither authors nor the advancement of science.
Moving forward, the research community needs to establish clearer guidelines about how reviewer assessments should evolve during the review process and what constitutes appropriate feedback for improvement. Conference organizers should consider implementing mechanisms that encourage more thoughtful initial reviews while providing better support for authors navigating conflicting feedback. The question worth watching is whether major conferences will develop more transparent and consistent review practices before the current system's unpredictability further erodes trust in scholarly communication.