March 31, 2026•1 min read•from Machine Learning

[D] Howcome Muon is only being used for Transformers?

Our take

Muon has gained significant traction in the realm of large language model (LLM) training, yet its application beyond Transformers remains largely unexplored. Despite its announcement highlighting a new training speed record for Cifar-10, searches for Muon in convolutional networks yield minimal results. This raises important questions about its scalability and effectiveness in other contexts. If faster training typically correlates with improved model performance, what factors are limiting Muon's broader adoption? Have key papers on its applications been overlooked? Let's delve into this intriguing topic.

Muon has quickly been adopted in LLM training, yet we don't see it being talked about in other contexts. Searches for Muon on ConvNets turn up basically no results, despite its announcement including a new training speed record for Cifar-10. In my experience faster training usually comes with better final models, so what's the deal? Does it not actually scale? Have I missed papers?

submitted by /u/lukeiy
[link] [comments]

Read on the original site

Open the publisher's page for the full experience

View original article →

Introducing AutoMuon, a one line drop in for AdamW [P]Hey everyone, I've been working on a small Python package called AutoMuon that makes the Muon optimizer usable as a drop-in replacement for AdamW in arbitrary PyTorch training pipelines. The core idea is relatively simple: Muon works primarily on 2D weight matrices (linear projections, conv layers) on hidden states, but you still need AdamW for embeddings, norms, and biases, etc. AutoMuon scans your model at init, figures out the right optimizer for each parameter automatically. I am open to PRs, especially for expanding the module-type exclusion list if you hit edge cases in your architecture. Would love to know if anyone tries it on something other than transformers or CNNs and what they find. I feel that it would likely struggle with fully custom architectures, like flash-linear-attention for instance, so that would require some user tuning. I am planning to add more tests for time series forecasting, genomics, language modeling, etc. I want to see how generalizable Muon really is! https://github.com/SkyeGunasekaran/automuon pip install git+https://github.com/SkyeGunasekaran/automuon.git submitted by /u/Skye7821 [link] [comments]

[D] Howcome Muon is only being used for Transformers?

Related Articles