The team behind Domyn has done something genuinely important here: they proved that a publicly funded, user-level supercomputer can train a trillion-parameter model without special privileges or proprietary infrastructure. That is not a modest claim, and the results back it up. Starting from a baseline where GPUs spent 68 of every 84 seconds waiting on the network, they iterated their way to 1,300 tokens per second per GPU across 384 GH200 accelerators, a 14x improvement. This is the kind of systems work that deserves attention, not because the numbers are flashy, but because every optimization they found went back upstream to PyTorch and TorchTitan.
What makes this story resonate is how it connects to the broader shift in AI infrastructure. We recently covered Master MoE Training with a Practical Megatron-LM Runbook, which laid out the operational playbook for mixture-of-experts training. The Domyn team lived that playbook under real constraints. They skipped pipeline parallelism to keep iteration cycles short, combined FSDP with expert parallelism confined to a node, and then systematically attacked the two real bottlenecks: inter-node bandwidth and host memory. Their fix for gradient accumulation, sending bf16 over the wire and summing in fp32 locally, is a textbook example of understanding where precision actually matters versus where it just costs bytes. Similarly, Bring GPU-accelerated nearest-neighbor search to Apple Silicon with mlxVS showed how squeezing performance from consumer hardware requires the same kind of thoughtful engineering. The domain changes, but the principle holds: respect your hardware's limits, then design around them.
The practical takeaway here is direct and actionable. If you are training large models on shared clusters, your first bottleneck is almost certainly the network, not the compute. The Domyn team's 84-second baseline step, with 68 seconds of network wait, is likely close to what many teams experience but do not measure. Their six lessons form a diagnostic checklist: check whether your GPU is idle while waiting for weights, whether your optimizer step is running on the wrong processor, whether your slowest rank is dragging the whole job down, and whether your memory allocator is hiding free memory from your compute stream. The last one is particularly insidious, a PyTorch caching allocator holding 21 GiB on side streams where the compute stream cannot touch it, causing OOM errors at 70% utilization, is the kind of bug that wastes weeks.
The open question that remains is the one they flagged themselves: group-limited routing. Making expert load perfectly even by constraining where tokens can go gave them a 7% throughput gain, but early runs suggest it may hurt training quality. That tradeoff is worth watching. If the quality cost proves real, the community will need a different approach to load balancing. If it does not, this becomes a standard technique for MoE training on bandwidth-constrained clusters. Either way, the Domyn team has given us a clear, reproducible path to frontier-scale training on accessible hardware. That is the kind of progress that moves the field forward, one upstream commit at a time.