The gap between offline metrics and real-world performance is the most honest story a systems paper can tell, and the researcher who posted this question is sitting on a strong one. A tiny median absolute deviation that still fails on a shelf-picking task is not a bug in the thesis. It is the thesis. The instinct to bury the failed deployment is understandable, but it is also the instinct that keeps the field stuck in a loop of polished benchmarks that do not survive contact with a robot arm. Include the failure. Frame it as the finding, not the flaw.
The likely culprit is exactly what the researcher suspects: tail errors and closed-loop compounding. A median deviation that looks negligible in a static comparison says nothing about the worst 1% of steps, and in a feedback loop one bad step can push the robot into a state where every subsequent step is worse. The original model might recover from a small misstep; the accelerated version might not, because its error distribution has a heavier tail. The fix is to report percentiles, not just the median. P99 and P99.9 of per-step deviation will reveal whether the accelerator is producing rare but catastrophic outliers. Add closed-loop success rate in simulation, then compare that to the real robot result. If simulation matches the real failure, the sim is doing its job. If it does not, the sim is part of the problem, and that is also worth publishing.
This is a moment where the field needs more negative results, not fewer. The PHP Patches Two Critical Flaws Four Days After Attackers Struck story shows what happens when systems are validated only under ideal conditions: the gap between expectation and reality shows up as an emergency. The same logic applies here. An accelerator that looks perfect offline but stumbles on a real task is a warning sign for anyone designing for deployment. The Focus on outcomes, not pricing models, when choosing AI for your spreadsheets piece makes a related point: what matters is how the tool behaves when you actually use it, not how it scores in a controlled test. And the Autonomous excavators from Bedrock Robotics dig up $4.1B valuation story is a reminder that real-world robotics is where the value lives, and where the failures are most expensive.
The concrete takeaway is this: submit the paper with the deployment result included, and structure the contribution around the discrepancy. The reviewer community at MLSys will respond to a paper that says "here is a metric that does not predict deployment, here is why, and here is a better metric." That is a contribution. A paper that says "my accelerator is fast and matches the median" is just another benchmark. The negative result is not a liability. It is the reason the paper should exist. The open question to watch is whether the community rewards this kind of honesty or punishes it. The answer will tell us a lot about whether systems research is ready to take robotics seriously.