The experiment proposed here is worth taking seriously, and not because it promises a flashy result. It asks a pointed question about a 50-million-parameter transformer trained on chess transcripts: when the model encounters moves that cannot happen, does its internal representation of the board break in ways that reveal what it actually understands? Karvonen's published model, probes, and intervention tools make this testable today. That is the kind of concrete, falsifiable inquiry that moves interpretability forward, and it deserves attention beyond a single thread.
What makes the proposal compelling is the gradient of violations. A pawn jumping to the center on Move 1 is not just improbable; it is structurally impossible. A Sicilian Defense with one move skipped is legal move-by-move but unreachable as a trajectory. A king under threat from an impossible angle is coherent square-by-square yet relationally broken. Each case targets a different layer of representation: movement rules, game path, piece geometry, piece identity, and strategic plausibility. If the model fails differently across these cases, then the latent board state is not a single monolithic thing. It is composed of distinct mechanisms that can be studied independently. If it fails the same way every time, that is also informative, but it would suggest the representation is shallower than the probes imply.
The practical implication for anyone working with language models is direct. If a model trained purely on character prediction develops a board state that can be read with linear probes, then the question of whether that state is causal or epiphenomenal is not academic. It bears on when we can trust a model's internal representations to guide behavior, and when those representations are just convenient fictions that happen to correlate with outputs. The proposed dissociation, coherent probes alongside disrupted prediction distributions, would be a genuinely new finding. It would show that the model computes attack geometry and piece relationships in ways that are separable from its next-token predictions. That would matter beyond chess.
The poster is right to note that the tools are public and the model is small enough to probe without a data center. That lowers the barrier to entry, and it means this is not a thought experiment. It is an experiment. The design is already sharp enough to run, and the failure signatures, probe coherence, continuation probability distributions, entropy, are measurable. We would like to see this run, and we would like to see the results published with the same transparency Karvonen showed. If the five cases produce qualitatively different failure modes, that is a map of the model's internal structure. If they do not, that is still a result, but it is a different one. Either way, the experiment is worth doing, and it is well within reach.