AdamW
AdamW at Beyond Market Intelligence is a file of 3 stories. The newest of them: “Tauon brings faster training and lower loss to AI optimization”, “How a tiny AI system learned to generate Bad Apple without a clock”, and “Tiered optimizer cuts MoE training memory demands by 97 percent”. A new optimizer called Tauon is turning heads in the r/MachineLearning community, and for good reason. Teaching a network to *remember* a video is one thing. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work… The list below is every AdamW story on Beyond Market Intelligence, newest first.
Tauon brings faster training and lower loss to AI optimization
A new optimizer called Tauon is turning heads in the r/MachineLearning community, and for good reason. It promises faster training and lower loss by rethinking how we handle polynomials and orthogonalization, building on ideas from Muon but streamlining the process. The initial results are compelling: Tauon hit a lower validation loss than both Muon and AdamW, all while running nearly as fast as the baseline. It's a small-scale test, but the stability and compute efficiency suggest real potential.
How a tiny AI system learned to generate Bad Apple without a clock
Teaching a network to *remember* a video is one thing. Teaching it to *dream* the whole thing forward from a single latent seed is another. This project ditches the timestamp entirely, letting a 16k-parameter recurrent core autonomously unroll 6,500 frames of Bad Apple from one initial state. What impresses me isn't the novelty of the architecture, it's the discipline. Noise injection, acceleration penalties, and a carefully scheduled rollout curriculum show a builder who understands that stability, not capacity, is the real bottleneck.
Tiered optimizer cuts MoE training memory demands by 97 percent
SkewAdam tackles the biggest pain point in MoE training head-on: optimizer state memory. By tiering precision across backbone, experts, and router, it cuts state from 50.6 GB to 1.29 GB, a 97.4% drop that lets a 6.78B MoE fit on a single 40GB GPU. That's not just incremental progress; it's a practical unlock for researchers stuck on smaller hardware. For deeper context on why memory efficiency matters, our coverage of Microsoft's TauGrid shows similar pressure points in AI workload management.