Sum of Separable Gaussians

Reimagining attention with a simpler, faster Gaussian approach

Scaled dot-product attention carries a heavy O(N²·d) burden because it scores every token against every other token. SSOG takes a different route: it learns a few Gaussian atoms per head and steers them geometrically from the query token. Because those atoms factor into a separable sum, complexity drops to O(N·√N·d). The results are telling. SSOG outperforms SDPA on CIFAR-100 and matches it on ImageNet while converging faster and using less memory. That is a practical step toward scaling attention without sacrificing quality.