The recent thread about ICLR's LLM feedback experiment captures something we've all felt at some point: the gap between what automated review promises and what it delivers. The user's experience, one or two genuinely useful points buried under three pages of nitpicking, is not a failure of the concept but a reminder of what these tools currently are. They are not replacements for judgment; they are tireless, sometimes clumsy assistants. The real skill is learning to separate the signal from the noise without dismissing the whole exercise. This is exactly the kind of practical friction that comes up when we discuss the mechanics of AI in Unlock LLM Training: A Practical Guide to Distributed Algorithms. The infrastructure is there, but the human interface is still being debugged.
We think the more interesting issue is the unspoken anxiety in the post. The user asks if the review stays public, and that question reveals a deeper concern about permanence and reputation. It's one thing to receive feedback; it's another to have it attached to your work indefinitely, especially when so much of it is trivial. This is where the conversation moves beyond the mechanics of prompt engineering and into the ethics of AI-assisted evaluation. If we are going to let these systems critique our work, we need clear rules about how that critique is stored, shared, and weighted. Otherwise, we risk creating a culture where authors game the system to avoid public nitpicking rather than using feedback to improve. That would be a step backward for the very transparency these initiatives claim to support.
The user's pragmatic take, that the process did improve the paper in some sense, is the most valuable insight here. It suggests that even imperfect feedback can be useful if you have the time and patience to sift through it. That is not a trivial condition. For a PhD student juggling deadlines or a researcher with multiple submissions, the cost of parsing three pages of edge-case nitpicks might outweigh the benefit. This is where the tool needs to evolve, not just in accuracy but in prioritization. We would tell anyone facing this situation to treat the feedback like a first-pass reviewer, not a final authority. Extract the structural issues, ignore the stylistic quibbles, and move on. The goal is not to satisfy the reviewer; it is to strengthen the argument.
The open question is whether the conference will treat this as a one-off experiment or a signal for how to refine the process. If the feedback remains public, authors deserve a warning, but they also deserve a mechanism to respond or flag irrelevant comments. That would turn a monologue into a dialogue. For now, the takeaway is simple: engage with the feedback, but do not let it derail your focus. Use it to catch the blind spots, not to chase the ghosts. The next iteration of this experiment will be far more telling, because it will show whether the organizers learned the difference between volume and value.