The numbers are real, and they matter. The ORL team has posted a 2,788 tokens-per-second benchmark on a single RTX 5090 using a diffusion language model, backed by a training method that learns to reduce uncertainty step by step rather than just supervising the final output. This is not a speculative roadmap or a slide deck promise. The weights, training code, and custom inference engine are all open source. For anyone who has felt constrained by the sequential bottleneck of autoregressive models, this is the first compelling evidence that diffusion-based text generation can match or exceed the throughput we expect from traditional LLMs, without the quality trade-offs that earlier attempts required.
What makes ORL interesting is not just the speed figure. It is the approach. On-Policy Residual Learning trains on fresh trajectories generated by the current model, replaying the exact residual states that appear during inference. That solves a mismatch that has plagued earlier diffusion LLMs: the distribution of residuals shifts during training, so models trained on stale offline data perform worse at inference time. The Adjacent Entropy Contraction loss then pushes each denoising step to reduce uncertainty relative to the previous step, rather than optimizing only the final output. The practical result is a model that finalizes more tokens per forward pass. On a QwenDLM-4B model, tokens per forward improved by 6.2x, and HumanEval pass@1 rose from 82.9% to 89.6%. That is not just a faster model; it is a smarter one.
The custom inference engine, dlmrt, deserves attention in its own right. By capturing the entire denoising loop in a CUDA graph and executing it entirely on the GPU, it eliminates the CPU launch and synchronization overhead that normally slows diffusion decoding. In single-H100 comparisons, it runs 1.2 to 1.6 times faster than SGLang or vLLM with identical models. Combined with support for NVFP4 inference on Blackwell GPUs, including the consumer RTX 5090, this means the performance gains are accessible on hardware that users can actually own. This contrasts with the usual pattern of benchmark-setting results that require clusters of data-center GPUs. The team also achieved 7,708 tokens per second on 8x B200 GPUs, but the single-card result is the one that changes the conversation.
We have seen a pattern in recent AI research where small, looped models outperform larger ones, as covered in Looping a small AI model beats a model over six times its size. ORL extends that principle into the diffusion space, and the open-source release means the community can test whether these gains hold across diverse workloads. The question the researchers pose to the community is the right one: is single-request latency the main bottleneck for local inference, or should future optimization focus elsewhere? For users running local models on consumer hardware, the answer is almost certainly latency. The task now is reproducing these results on different prompts, measuring real-world variability, and watching whether the community can match the 2,788 tokens-per-second claim with their own RTX 5090. If they can, the next wave of local AI tools will look very different from the ones we use today.