The question of why neural networks become unreadable to each other once they've been trained independently has haunted weight-space learning for years. The usual suspect is symmetry, permute a few hidden units, flip some signs, and you've got a different parameter vector that still computes the same function. But as this new study on roughly 1.8 million fitted SIRENs makes clear, there's a meaningful difference between saying symmetry exists and proving it explains the observed collapse. The author separates the claims cleanly: yes, the symmetry group is there; yes, accounting for it helps; but no, that doesn't automatically mean symmetry is what's driving the degradation between shared-initialization and independently fitted networks. That distinction matters, and it's one most papers in this space never bother to draw.
The headline result is striking. By randomizing only the exact symmetry group while keeping each network's represented function fixed, the study reproduces 79.1 of the 80.4 accuracy points in the MNIST shared-init versus random-init gap. That's not a correlation, it's a controlled intervention that isolates the symmetry scatter as the causal mechanism for nearly all the degradation. Sign flips account for roughly 63 points, neuron relabeling about 15, and integer phase shifts about 1. But the study carefully notes that this establishes sufficiency, not mediation. The naturally occurring gap might be driven by other factors that just happen to look like symmetry from the outside. That's a subtle but essential caveat, and it's the kind of intellectual honesty that makes this work worth taking seriously. It also connects to a broader pattern we've been tracking in our own coverage, like how Accelerate Local LLM Learning: A New Prototype for Faster Fact Correction shows that fixing a model's internal representations often requires more than just tweaking the loss function, you have to understand what the weights actually encode.
The deeper implication, though, is computational rather than informational. A complete invariant, one that quotients out the symmetry group entirely, is informationally equivalent to just querying the function itself. And that's exactly what the results show: when you FLOPs-match weight-space inference against simply querying the INR as a function, the function-space route wins decisively, hitting 95.3% accuracy at 1.6 MFLOP versus 64.4% at 5.5 MFLOP for the best weight-space rung. So if you can always just call the function, why bother with weight space at all? The answer isn't obvious, and the study's willingness to pose that question openly is refreshing. It's the same spirit we saw in [CABiNet (ICRA 2021) vs YOLO26-sem on UAVid: accuracy, compute, and GPU latency [P]](/post/cabinet-icra-2021-vs-yolo26-sem-on-uavid-accuracy-compute-an-cmtkekw0v01otrged4e0nrz3u), where the authors measured trade-offs rather than assuming newer meant better. Here, the trade-off isn't just accuracy, it's the entire justification for operating in weight space when function space is simpler and cheaper.
What we'd tell a reader asking whether this changes how they should think about weight-space learning is this: the burden of proof has shifted. If you're building a model that reads weights directly, you need a computational argument for why that's better than just querying the function, because the informational case is now dead. The one-hidden-layer identifiability proof is a nice theoretical anchor, but the real test will come from whether anyone can break the depth-two invariants or find a counterexample to the maximality argument. That's the open question to watch, not because the answer will be clean, but because the study has done the rare thing of making its own failure modes explicit. We'd rather see someone try to kill it than let it stand on untested assumptions.