NeurIPS

Navigating peer review shifts: strategies for uncertain NeurIPS outcomes

A reviewer dropping a score after you've already addressed most of their concerns is frustrating, especially when no new justification appears.

4 min readMachine Learning

There's a particular kind of frustration that comes with watching a score drop after you've done the work. The researcher in that thread did everything right: they addressed three of the four concerns raised in initial reviews, they engaged in good faith, and they waited. Then the reviewer quietly lowered their score without a new justification. No new evidence, no fresh critique. Just a silent step backward. We've all been there, and it stings because it breaks an unspoken contract: that the review process is a dialogue, not a moving target. The real question isn't about the score itself. It's about whether the Area Chair is paying attention.

This is where the human element of research pipelines becomes the deciding factor, and it's worth exploring why. The question is whether an AC can advocate for a paper when reviewers are lukewarm or inconsistent. The honest answer is: sometimes. An engaged AC reads the meta-review, notices when a reviewer's shift lacks justification, and pushes back. A silent AC, as described here, leaves authors in limbo. That's not a technical problem; it's a communication one. And it's the same challenge we see in Exploring Paragraph Structure: How LLMs Navigate Token Space, where the real signal isn't in the individual tokens but in how they're arranged. Similarly, the signal in a review isn't just the number. It's the narrative the reviewers and AC build together. If that narrative breaks down, the researcher is left guessing.

So what do we tell the researcher who's staring at a lowered score and a silent AC? First, don't assume the AC is ignoring you. They're likely juggling dozens of papers, and the meta-review is their only window into your exchange. But here's the practical move: make their job easier. If the AC has to dig through the discussion thread to see that you addressed the weaknesses, they might miss it. Write a concise, direct summary of each rebuttal point and where you addressed it. Think of it as a map, not a memoir. This is the same logic behind Unlock LLM Training: A Practical Guide to Distributed Algorithms: complexity is manageable when you break it into clear, actionable steps. The same applies to your meta-review response. You're not just defending your work; you're guiding the AC to the evidence.

Second, and this is the part that often gets missed: a lowered score after a rebuttal isn't always a judgment on your paper. Sometimes it's a reviewer who felt pressured to engage, then got defensive. That's not your failure, but it's also not something you can control. What you can control is whether you've given the AC a reason to advocate for you. In our experience, ACs are more likely to push back on a reviewer when they have a clear, documented response that addresses the core concerns. That's why we'd tell the author to focus less on the AC's silence and more on the clarity of their own final response. The outcome may not change, but the process becomes less opaque. And for anyone navigating this, we'd suggest a quick read of Unlock ChatGPT for Work: A Practical Guide to Getting Started for a reminder that structured, direct communication often outperforms broad, unfocused effort.

The uncomfortable truth is that peer review is a human system, and humans are inconsistent. But that doesn't mean you're powerless. The next time a reviewer drops a score without explanation, treat it as a signal to sharpen your own case, not to panic. The AC may be silent, but that doesn't mean they're not listening. The real takeaway here is simple: your job isn't to win over every reviewer. It's to make the AC's decision easier. Do that, and you've done your part.

From Machine Learning

So our paper had very good initial reviews but one of the reviewers decreased now their score although we addressed 3 out of 4 weaknesses. There’s no further justification or something like “your results arise more issues”. It seems to be very annoying because why decreasing now and not having assigned the lower score beforehand. I wanted to ask to people that was accepted previously with “middle” scores from reviewers (avg 3.5 for example), because I guess that in those cases AC helped to push up the scores. Did you focus more on the meta review? Was your AC talkative…

Read the original at Machine Learning