peer review
peer review on Beyond Market Intelligence: a running collection of 34 stories we have gathered and hand-picked because they are worth your time. Every post here touches on peer review in some way — the news, the analysis, the deep dives, and the occasional surprise find. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work with data. New stories are added to this page as we find them, so check back if you want to keep up with what is happening around peer review, or subscribe to the RSS feed to get them as soon as they are published. Browse the collection below, or head back to the homepage to see everything Beyond Market Intelligence is covering right now.
I regret reviewing for AAAI [D]
Reviewing for prestigious conferences like AAAI can feel like a significant time investment, particularly when reciprocity isn’t guaranteed. A recent Reddit post articulated a common sentiment: the allure of feeling valued can outweigh the practical realities of dedicating time to evaluating work that doesn’t directly benefit one's own submissions.
ACML 2026 Journal Track Any update ?[D]
Submitting to ACM’s 2026 Journal Track can be a pivotal step in research dissemination. Many researchers, like /u/Jealous_Key_4030, are awaiting review decisions—the official release date was August 27th, and timely feedback is crucial. If you've received your review, please share your experience to help others. Delays can be frustrating, so reaching out to the program chairs is a proactive approach. For further insights into presenting machine learning work, consider "Good Machine Learning Posters," a related discussion exploring effective poster design.
*ACL Findings or TMLR? [D]
Navigating the conference publication landscape presents a strategic challenge. With NeurIPS appearing unlikely given current scores, the decision between Transactions on Machine Learning Research (TMLR) and *ACL Findings* warrants careful consideration. While both venues offer visibility, *ACL Findings* likely presents a higher probability of acceptance. Genuinely curious about industry perspectives: would you prioritize *ACL Findings* or TMLR on your publication record? For deeper insights into related AI discovery research, explore our article on "Autonomous Mathematical Discovery in an Open-World Multi-Agent Environment."
Is EMNLP not going to Provide a MetaReview [D]
A concerning trend has emerged within the NLP community: the absence of meta-reviews following EMNLP decisions. Unlike ACL, EMNLP has not publicly provided these crucial evaluations, leaving submitters in the dark regarding the rationale behind accept/reject outcomes. One user, facing a situation where an Area Chair’s recommendation for acceptance was overridden by reviewers, is questioning whether low reviewer scores influenced the decision. This uncertainty complicates decisions about resubmission and potential ARR cycles.
Archival vs non archival workshop [R]
Understanding NeurIPS workshop archiving is crucial for maximizing the impact of your work, particularly for graduate school applications. A key distinction exists: NeurIPS workshops, like many others, are typically non-archival. Consequently, publication in a proceeding may carry less weight than a peer-reviewed journal. For context, consider how preprints and subsequent publications are handled—a discussion explored in our article, "How to cite/talk about preprint-subsequent works for a camera-ready version?". Prioritize venues that offer robust archival to strengthen your academic record.
How to cite/talk about preprint-subsequent works for a camera-ready version? [R]
Navigating citations when a paper transitions from preprint to a conference camera-ready can be nuanced. To maintain both novelty and acknowledge impactful subsequent work, consider citing your preprint initially, then clearly state it's the precursor to the current publication. Acknowledge any works building upon your preprint’s methodology, demonstrating its influence. This approach transparently reflects the research lineage. For further insights into related challenges in AI research integrity, explore "AAAI 2027 Reviewer Bidding and Assignment Integrity [D]" for a deeper understanding of evolving ethical considerations.
acl arr august 2026 (desk rejected ) [D]
Experiencing a desk rejection from ACL citing prior review, despite never submitting the work, is understandably frustrating. It suggests a potential issue with reviewer records or a possible mix-up. While a definitive solution is challenging without further investigation, carefully review submission logs and consider contacting the ACL program chairs with detailed documentation of your submission history. This situation highlights the importance of understanding conference archival policies, as discussed in our related article, "Archival vs non archival workshop."
ICLR numbered citations possible? [R]
Navigating citation formatting for ICLR submissions can be a critical detail. The instructions specify Author Year format, but a shift to numbered citations (without spaces) risks immediate desk rejection. While community experience on this is valuable, definitive guidance remains scarce. Submitting with a non-compliant format introduces unnecessary risk. For further context on navigating conference deadlines and related considerations, explore our article, "NeurIPS 2026 Author Notifications Close to ICLR Deadline." Prioritize adherence to the provided guidelines to ensure your submission’s review.
NeurIPS 2026 Author Notifications Close to ICLR Deadline [D]
NeurIPS 2026 author notification deadlines—September 24th—are fast approaching, coinciding closely with the ICLR submission deadline. A common concern arises: are extended Area Chair and reviewer discussion phases typical? Many authors report frustration when rebuttals go unaddressed. Given this timing, researchers are strategically evaluating ICLR submissions as a contingency. As one example, our recent article, "Mathematical Experiments Are Becoming Abundant Through Human-Machine Teaming," explores related challenges in rigorous experimentation. Good luck navigating these crucial deadlines!
AC comment and our reply disappeared on OpenReview [D]
A concerning issue has emerged on OpenReview: several users report that an AC's initial comment and the author's subsequent reply have vanished. This comment, crucial for understanding reviewer feedback and addressing concerns, summarized key questions and weaknesses. The disappearance raises questions about transparency, particularly if it obscures the rationale behind potential rejection decisions. We've encountered similar discussions around data management challenges, as explored in our article, "A Day in the Life of a Data Scientist in 2026."
How much does adding an honest limitations section hurt the paper? [D]
Addressing limitations honestly in research papers—while generally beneficial—raises critical questions about reviewer bias and potential requests for remediation. Does openly acknowledging constraints negatively impact perception, or will reviewers demand fixes outlined in the limitations section? Furthermore, the introduction of AI reviewers introduces a novel consideration: could these limitations inadvertently bias algorithmic assessment? Exploring these nuances, as discussed in "My Model Was Cheating on Its Own Test," highlights the complexities of transparency in AI research.
For the people who got reviews back from neurips, cvpr, eccv, etc and also tested their paper through an agentic reviewer like the stanford one, how different were the reviews? [D]
For those who recently received reviews from NeurIPS, CVPR, ECCV, or similar conferences, and also utilized agentic reviewer tools like the Stanford model, a compelling question arises: how do the reviews compare? We're exploring the divergence between human and LLM assessments, seeking insights into this evolving landscape. Early indications suggest significant variations, prompting a deeper understanding of how AI-assisted review impacts the peer review process. For further context on related challenges, see our article, "My Model Was Cheating on Its Own Test."
NeurIPS 2026 Main Track — Theory papers score tracking post Rebuttal [D]
Following the NeurIPS 2026 rebuttal period, a discussion has emerged regarding Theory paper score distributions. Early reports suggest scores may be trending lower across disciplines this year. To facilitate a clearer understanding of the landscape, authors are invited to share their scores (x/x/x), confidence levels (x/x/x), and whether scores shifted post-rebuttal, optionally specifying the broad area of research. One author reported a 4/4/4 score with 3/3/3 confidence.
Completely dead NeurIPS review period from both ends? [D]
A concerning trend has emerged during the NeurIPS review period: unusually prolonged silence from both reviewers and authors. Reports indicate reviewers abandoning the process post-initial reviews, while authors remain unresponsive even without submitting rebuttals. This phenomenon, observed across multiple submissions, raises questions about shifts in academic publishing practices—potentially a strategy of widespread submission with limited follow-through. As noted in a related discussion on score tracking post-rebuttal for theory papers, NeurIPS 2026, maintaining engagement throughout the review cycle remains crucial.
NeurIPS 2026 post-rebuttal score distribution poll [D]
Curious about the NeurIPS 2026 post-rebuttal score distribution? With discussions surrounding potentially lower scores this year, a quick poll aims to gauge the average score breakdown after the rebuttal phase—excluding confidence weights. This is a preliminary look, acknowledging inherent self-selection bias. Share your vote here: [https://loppy.be/poll/yczuv8yo](https://loppy.be/poll/yczuv8yo). For deeper insights into NeurIPS trends, see our related article, "NeurIPS 2026 Main Track — Theory papers score tracking post Rebuttal [D]," for specific analysis.
Do ACs also give scores? [D]
Navigating NeurIPS submissions can be confusing, especially for first-timers. Many authors wonder if Area Chairs (ACs) provide scores during Phase 2, the author-reviewer discussion. While you've received your meta-review, the absence of direct AC comments is a common query. It’s standard for ACs to remain largely silent during this phase, focusing on guiding the discussion. For more on navigating conference commitments, see our article, "Missed EMNLP commitment deadline, what can be done?". Focus on addressing reviewer concerns and refining your paper.
The Downsides of LLM-Generated Peer Reviews [D]
The increasing use of Large Language Models (LLMs) in peer review presents notable challenges. Primarily, LLMs struggle to prioritize concerns, often generating an endless list of technically possible but practically insignificant variables that overwhelm authors. Secondly, reviews frequently become overly abstract, criticizing entire research fields instead of specific methods. This lack of detail, coupled with a tendency to equate superficial terminology with substantive similarity, diminishes the value of the review process.
No rebuttals from neurips authors [D]
Many NeurIPS authors are experiencing frustration with a lack of reviewer responses, a sentiment echoed in recent discussions. It appears the absence of author rebuttals is surprisingly common; a significant number of submissions, including borderline papers with positive Area Chair feedback, haven't received them. This leaves authors understandably perplexed. While challenging, this situation highlights a broader issue within the peer review process. For deeper insights into related concerns, explore our article, "neurips 2026: ACs and reviewers have disappeared."
Bad but typical NeurIPS experience? [D]
The NeurIPS review process, as highlighted by one researcher's experience, can be a frustrating lottery. Despite conscientious reviewing and generous scoring, unexpectedly harsh reviews and unresponsive area chairs created a deeply discouraging experience. Adversarial reviewer feedback, coupled with a late-stage AC response, underscored the system’s inherent unpredictability and potential toxicity. This highlights a broader issue within the AI research community, prompting discussions around reviewer accountability—as explored in articles like "NeurIPS 2026: If the rebuttal addresses your concern, please raise your score."
NeurIPS 2026: If the rebuttal addresses your concern, please raise your score [D]
A persistent challenge within the NeurIPS community involves reviewer scoring discrepancies: concerns adequately addressed in rebuttals are not always reflected in adjusted scores. We urge reviewers to align scores with the resolution of stated concerns, regardless of personal methodological preferences. Scientific exploration thrives on diverse perspectives, and valuing rigorous responses strengthens the peer-review process. As explored in "Coding Agents Don’t Need Bigger Context Windows — They Need a Context Compiler," a focus on efficient context management is key to progress.
Conference Reviews: Asking Too Much? [D]
Conference reviews sometimes request additions that extend beyond a paper's page limit, a practice particularly prevalent at top-tier events. While these expansions can be valuable, they often better suit journal publication—a concern that recently led one author to retract a submission. Does this approach inadvertently hinder future journal opportunities? We invite discussion on whether such additions are a conference review quirk or a sign of a broader disconnect.
Question about NeurIPS discussion phase [D]
Navigating the NeurIPS discussion phase can be unpredictable. A common question arises: how often do reviewers update scores after indicating concerns are resolved? Experience suggests it’s less frequent than one might hope, particularly when initial engagement is limited. You're not alone in observing this—others have noted similar patterns. Our community has explored this dynamic further in "Conference Reviews: Asking Too Much?" As your case demonstrates, persistence can yield results, ultimately leading to score adjustments.
Neurips 2026: does every metareview recommend accept/reject? [D]
Navigating NeurIPS decisions can be perplexing. A recent discussion reveals a surprising trend: some metareviews already include an accept/reject recommendation—often a rejection. Your team’s experience, with a metareview expressing cautious optimism despite reviewer disengagement, highlights this complexity. While a strong rebuttal is crucial, the lack of engagement raises questions about future prospects. As explored in "No rebuttals from NeurIPS authors," reviewer responsiveness remains a significant challenge. Consider carefully whether continued hope aligns with the current situation.
neurips 2026: ACs and reviewers have disappeared [D]
NeurIPS 2026 reviewers and Area Chairs have inexplicably vanished, leaving several submitters in a state of uncertainty. Early rebuttal submissions, made via the designated "Rebuttal" button before the official discussion period opened, appear to have triggered no notifications, a problem also experienced by reviewers. Despite attempts to utilize meta-comments, reminders, and direct PC contact, responses remain absent with only one day left in the review cycle. This situation, impacting potential oral and spotlight candidates, highlights a systemic issue.