Rebuttals Can't Fix a Reviewer Who Moves the Goalposts

Navigating the post-rebuttal landscape of ICML average scores can be frustrating, especially when unexpected reviewer comments arise.

3 min readMachine Learning

The reviewer who handed out a 2 after borrowing a concern from another reviewer, one that the original reviewer had already dismissed, isn't making a fair critique. They're moving the goalposts, and no amount of rebuttal writing can fix that. This isn't about the quality of the work anymore. It's about a process that allows one inconsistent voice to override the substance of the review itself.

Here's what this means for you in practical terms: your rebuttal isn't just a defense of your methods or results. It's also a record of the reviewer's own inconsistency. When you encounter a point that was raised, then retracted by one reviewer, and then recycled by another as a fresh objection, you're not obligated to treat it as a new problem. You can point out the contradiction directly. You can say, "This concern was already addressed and dismissed in the prior review cycle. We're happy to clarify further, but we won't treat a settled issue as a new one." That's not defensive. That's precise. And it's the kind of precision that helps editors see what's actually happening.

The frustration with Paper Co-Pilot's scoring, where 4.2 supposedly lands in the top 40%, only adds to the sense that the system is working against you, not with you. But here's the thing: that tool isn't the final arbiter of your contribution. It's a heuristic, and a blunt one at that. The real damage isn't the number. It's the emotional tax it takes on researchers who are already doing the hard, careful work. You can't control the reviewer's mood or the tool's calibration. You can control how you document the exchange, and you can make sure that your rebuttal stands as a clear, factual account of what was said, what was resolved, and what was simply repeated.

So don't chase the goalposts. Write your rebuttal as if the editor is reading it cold, without the baggage of the original review. State plainly that the issue was raised, addressed, and dismissed by the very person who originally raised it. Then move on. If the editor sees the contradiction, they'll act on it. If they don't, no amount of additional arguing will change their mind. But at least you'll know you did your part, clearly, honestly, and without letting a moving target dictate your next move.

From Machine Learning

I have an average of 3.5. One of the reviewer gave us a 2 by bringing up a new issue he hadn't mentioned in his initial review, taking that from another reviewer's concerns. The reviewer he took it from already mentioned that it isn't an actual issue too.

Paper Co-Pilot is driving me crazy, apparently 4.2 is just the top 40% of papers according to it.

Read the original at Machine Learning