MoE
MoE on Beyond Market Intelligence: a running collection of 3 stories we have gathered and hand-picked because they are worth your time. Every post here touches on moe in some way — the news, the analysis, the deep dives, and the occasional surprise find. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work with data. New stories are added to this page as we find them, so check back if you want to keep up with what is happening around moe, or subscribe to the RSS feed to get them as soon as they are published. Browse the collection below, or head back to the homepage to see everything Beyond Market Intelligence is covering right now.
Continued development of the model based on the SSN [D]
After a six-month hiatus, development continues on Project NORD, now evolving into NORD 5.5 — Flash, a CPU-first spiking language model. This iteration prioritizes architectural simplification, moving away from artificial time dimensions and focusing on causal processing with innovations like sparse MoE and persistent memory banks. The core objective is to optimize for CPU inference, a significant shift from previous iterations. Initial benchmarking will compare NORD 5.5 against earlier versions, evaluating performance metrics like perplexity and RAM usage.
![SkewAdam: A tiered optimizer that cuts MoE state memory by 97% (fits a 6.7B MoE on a 40GB GPU) [R]](https://preview.redd.it/1457xi9fcqeh1.jpg?width=140&height=90&auto=webp&s=879aad6df9e51a2735d91112d01518ff76ba3cbe)
SkewAdam: A tiered optimizer that cuts MoE state memory by 97% (fits a 6.7B MoE on a 40GB GPU) [R]
Introducing SkewAdam, a novel tiered optimizer designed to dramatically reduce memory consumption in Mixture-of-Experts (MoE) training. Research demonstrates a remarkable 97% reduction in optimizer state memory—allowing a 6.7B MoE model to comfortably fit on a single 40GB GPU. SkewAdam intelligently allocates precision based on parameter behavior, optimizing backbone, expert, and router components. See the full details and code on arXiv and GitHub. For broader context on advancements in AI hardware, explore "What to watch for after Jensen Huang’s Japan visit."
Tried testing qwen 35b moe model on s26 ultra , without compromising on precision [R] ,[D]
Early testing reveals promising results for running a private Qwen 35B MoE LLM on an S26 Ultra, demonstrating a potential for approximately 90 tokens/second input processing and 8 tokens/second output generation after optimization. This achievement, realized through self-directed AI/ML exploration and leveraging available compute resources, highlights the accessibility of advanced model deployment. The author, without disclosing implementation details, is actively seeking collaborators to further test and refine this mobile runtime.