How much of the weight-space perception gap is actually symmetry? Evidence from ~1.8M fitted SIRENs [R]
Our take
The recent exploration into weight-space learning, detailed in Improved compression of Bad Apple into a Neural Network, has yielded some fascinating and potentially paradigm-shifting results. The author, u/ITheClixs, tackles a persistent puzzle: why does direct interpretation of neural network weights – a burgeoning area of research – work surprisingly well when networks share initializations, yet falters dramatically when trained independently? The conventional wisdom attributes this to parameter symmetry—the idea that different weight configurations can represent the same function. However, the author meticulously dissects this concept, demonstrating that the degradation isn't solely attributable to permutation and sign flips, but also to less-appreciated symmetries related to integer-pi phase transformations. This nuanced understanding, coupled with a rigorous mathematical framework and extensive empirical validation using SIREN-style implicit neural representations, represents a significant advancement in our ability to reason about and potentially control weight-space semantics. The sheer scale of the experiment—nearly 1.8 million fitted networks—lends considerable weight to the findings.
The author’s key discovery—that randomizing the symmetry group alone accounts for a staggering 79.1% of the accuracy degradation between shared and independent initialization—is particularly compelling. It suggests that the intuitive notion of “random” weights is far more structured than we often assume, and that these underlying symmetries play a much larger role in determining the interpretability of weight spaces. Furthermore, the comparative analysis of different approaches to extracting information from these weights, including direct quotienting of the symmetry group and orbit-valued reframing, illuminates the trade-offs between computational efficiency and representational power. This builds upon previous explorations into neural network compression, like the work showcased in I Compressed Bad Apple into a 3MB Neural Network, which also demonstrated impressive results using SIREN networks, albeit with a different focus. The comparison of weight-space inference against directly querying the realized function is particularly insightful, hinting that the justification for operating directly in weight space might ultimately be rooted in computational advantages rather than inherent informational superiority.
This research underscores a crucial point about the evolving landscape of AI: the increasing importance of understanding the *mathematical structures* underlying neural networks, rather than simply treating them as black boxes. The author’s work highlights the power of combining theoretical analysis—the derivation of exact cross-layer invariants and the distributional Fourier transform—with large-scale empirical validation. The detailed documentation, including the repository containing the paper, code, and experimental results, is a model for open and reproducible research. It's clear that the ability to manipulate and interpret neural network weights—to "read" the knowledge encoded within—is not merely a desirable feature, but a fundamental requirement for building more explainable, controllable, and efficient AI systems. The spectral neuron concept, explored in The spectral neuron - an ML primitive for scalable and interpretable models, offers a complementary approach to achieving interpretability, and the interplay between these methods warrants further investigation.
Looking ahead, a key question emerges: if a complete invariant representation is indeed informationally equivalent to direct function access, how do we reconcile this theoretical equivalence with the observed computational advantages of weight-space inference? The author rightly cautions against attributing causality solely to symmetry, but the overwhelming impact of symmetry scatter on the observed degradation suggests that it’s a critical factor to consider. Future research should focus on developing techniques to explicitly exploit these symmetries—perhaps through specialized architectures or training algorithms—to unlock the full potential of weight-space learning while minimizing the pitfalls of independent initialization. The author’s call for scrutiny and attempts to break the invariants is a testament to the rigorous spirit of scientific inquiry, and we eagerly anticipate further developments in this exciting area.
I’ve been looking at a fairly basic question in weight-space learning that I don’t think gets separated cleanly enough:
Why does reading semantics directly from neural network weights work pretty well when the networks share an initialization, but collapse when the networks are fitted independently?
The usual explanation is parameter symmetry. Permute hidden units, flip equivalent signs, etc., and two parameter vectors can represent the same function while looking completely different to a downstream model.
But there are actually several different claims hiding in that explanation:
the parameterization has a symmetry group,
accounting for that symmetry improves weight-space prediction,
the symmetry is actually sufficient to explain the observed degradation between shared-init and independently fitted networks.
Those aren’t equivalent, so I tried to measure them separately.
The setting is SIREN-style implicit neural representations.
For a hidden sine neuron, the relevant function-preserving transformations generate the infinite dihedral group
D_inf = Z semidirect_product Z_2
and including neuron permutations gives the layer action
D_inf wr S_n.
For one hidden layer, I prove generic identifiability modulo this group using the distributional Fourier transform of the realized function.
Roughly, the Fourier transform becomes an atomic measure supported at the incoming frequencies +/- w_i, which lets you recover the parameters up to exactly the D_inf wr S_n action under explicit genericity conditions.
One consequence is that this isn’t just the usual permutation/sign story. Integer-pi phase transformations are affine rather than linear, so they aren’t captured by symmetry descriptions restricted to monomial matrix actions.
At depth two things get more annoying because a neuron’s outgoing weights are simultaneously acted on by the next layer. I ended up constructing exact cross-layer invariants by coupling the layers through the second-layer Gram matrix instead of treating neurons independently.
The empirical part then uses roughly 1.8 million fitted INRs across MNIST, FashionMNIST, and CIFAR-10, with controlled protocols separating shared initialization, optimization stochasticity, and independent initialization.
The result I found most interesting:
Randomizing only the exact symmetry group, while keeping each network’s represented function fixed, destroys 79.1 of the 80.4 accuracy points in the MNIST shared-init vs. random-init gap.
I want to be careful about the interpretation here.
This establishes sufficiency: symmetry scatter alone can reproduce almost the entire degradation.
It does not establish that 79.1 / 80.4 of the naturally occurring gap is causally mediated by symmetry. Those are different estimands.
Breaking the group apart, sign flips account for roughly 63 points of that induced loss, neuron relabeling about 15, and integer phase shifts about 1.
There was another result that changed my interpretation of the problem quite a bit.
A reader that directly quotients the D_inf wr S_n structure on the raw parameters reaches 0.917, compared with:
0.628 for the best orbit-valued reframing,
0.526 for the same reader family over a fixed invariant encoding,
0.265 for a permutation-equivariant baseline.
But when I FLOPs-match weight-space inference against simply querying the INR as a function, the function-space route is still much better:
95.3% at 1.6 MFLOP using 64 learned query coordinates
versus
64.4% at 5.5 MFLOP for the best weight-space rung on that frontier.
That leads to what I think is the more interesting conceptual question:
If a complete invariant is informationally equivalent to access to the realized function, then the strongest justification for operating directly in weight space may ultimately have to be computational rather than informational.
Everything is public here:
https://github.com/ITheClixs/project-siren-gap
The repo includes the paper, implementation, tests, pre-registrations, lab notebook, prediction ledger, claims ledger, and experimental results.
I’d particularly appreciate criticism on three things:
whether the sufficiency/mediation distinction is being drawn correctly,
whether anyone sees a counterexample or missing assumption in the one-hidden-layer maximality argument,
whether there is related work on affine symmetry groups of periodic-activation networks that I’m missing.
Also very interested in attempts to break the invariants or reproduce the group-randomization result.
If something here is wrong, I’d rather find out from someone trying to kill it.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience