Shampoo

Why Adam Misses What Gradient Descent Naturally Preserves

The loss landscape of factored models hides a subtle trap, and this work exposes it with refreshing clarity. Adam's per-coordinate scaling breaks rotation invariance, a property GD naturally respects, and that single flaw explains why it loses low-rank bias. The one-parameter transition to a shared scalar is a clever mechanism, proving anisotropy, not adaptivity, is the culprit. Muon's crossover near 4% tail energy is particularly striking, showing both claimed behaviors along one axis.