Rewriting model inference with CUDA kernels: the bottleneck was not just GEMM [P]
Our take
In the ever-evolving landscape of machine learning and artificial intelligence, the quest for optimization remains a central theme for developers and researchers. The recent article on rewriting model inference with CUDA kernels underscores a critical insight: the bottleneck in small-batch runtime performance isn't merely about slow General Matrix Multiply (GEMM) operations, but rather about the inefficiencies in the surrounding infrastructure. This perspective resonates with our ongoing discussions about innovative solutions to common challenges in AI, similar to the insights shared in articles like Released a free 9.8M doc Indic multilingual corpus — Hindi, Bengali, Tamil, Telugu + 7 more (CC0, HuggingFace) and Witchcraft, fast local semantic search on top of SQLite — both of which emphasize the importance of rethinking traditional methods to unlock greater efficiency and capability.
The author’s approach of using C++/CUDA to rewrite the model inference path directly is particularly noteworthy. This method reflects a growing trend among developers to move beyond generic frameworks like PyTorch or TensorRT, which may not be optimized for specific use cases, particularly in real-time machine learning scenarios where batch sizes are typically one. The findings highlight that latency is not just a factor of mathematical computation, but also of the fragmented small kernels, layout transitions, and the overhead from quantization and dequantization. This is a crucial point for teams working on AI applications in robotics, autonomous systems, and advanced machine learning tasks, where every millisecond counts.
Moreover, the revelation that lower precision does not automatically translate into performance gains challenges long-held assumptions in the field. The nuanced understanding of floating-point precision—where FP8 may yield consistent benefits while FP4 can be mixed—encourages developers to critically assess their optimization strategies. This aligns with our broader narrative about the need for a deeper understanding of technology as we push towards more sophisticated applications in AI. As discussed in the No new paper under review in TMLR since May 09?#tab-under-review-submissions#tab-u) article, the demand for innovation in AI frameworks is palpable, and this exploration into CUDA-based inference may signal a shift in how we approach model optimization.
The implications of this work extend beyond immediate performance enhancements. As more developers adopt similar strategies, we could witness a shift in the tools and methodologies employed across the industry. Rethinking the inference pipeline could lead to more tailored solutions that enhance productivity and responsiveness, especially in real-time applications. The author’s challenge to reconsider when to utilize generic compilers versus custom optimizations is a crucial dialogue that could shape future research and development paths.
As we look ahead, it is essential to consider how these insights will influence the broader AI ecosystem. Will we see a shift towards more customized, performance-oriented approaches to machine learning model deployment? The ongoing exploration of CUDA and similar technologies may very well pave the way for transformative changes in the efficiency and effectiveness of AI applications. This is a space worth watching closely, as the potential for innovation continues to unfold.
I’ve been working on a CUDA-first inference runtime for small-batch / realtime ML workloads.
The core idea is simple: instead of treating PyTorch / TensorRT / generic graph runtimes as the main execution path, I rewrite the model inference path directly with C++/CUDA kernels.
This started from robotics / VLA workloads, but the problem is more general.
In small-batch inference, the bottleneck is often not just a single slow GEMM. A lot of latency comes from the runtime glue around the math:
- fragmented small kernels
- norm / residual / activation boundaries
- quantize / dequantize overhead
- layout transitions
- Python / runtime scheduling
- graph compiler fusion failures
- precision conversion around FP8 / FP4 regions
For cloud LLM serving, batching can hide a lot of this.
For robotics, VLA, world models, and other realtime workloads, batch size is usually 1. There is nowhere to hide. Every launch, sync, and format boundary shows up directly in latency.
Some current results from my implementation:
| Model / workload | Hardware | FlashRT latency |
|---|---|---|
| Pi0.5 | Jetson Thor | ~44 ms |
| Pi0 | Jetson Thor | ~46 ms |
| GROOT N1.6 | Jetson Thor | ~41–45 ms |
| Pi0.5 | RTX 5090 | ~17.6 ms |
| GROOT N1.6 | RTX 5090 | ~12.5–13.1 ms |
| Pi0-FAST | RTX 5090 | ~2.39 ms/token |
| Qwen3.6 27B | RTX 5090 | ~129 tok/s with NVFP4 |
| Motus / Wan-style world model | RTX 5090 | ~1.3s baseline → targeting ~100ms E2E |
The Motus / world-model case is especially interesting.
The baseline path is around 1.3s end-to-end. The target is ~100ms E2E, but the hard part is not simply “use a faster GEMM”. The bottlenecks are VAE, joint attention, launch fragmentation, and a large amount of glue around the actual math.
One lesson from this work: lower precision is not automatically a win.
FP8 has been consistently useful. FP4 / NVFP4 is more mixed. It can help memory footprint and some large GEMM regions, but if the FP4 region is small, discontinuous, or surrounded by conversion / scaling overhead, the end-to-end speedup can be tiny.
For example, in some VLA / world-model paths, FP4 over FP8 only gives a few percent latency improvement unless the region is large and deeply fused.
This changed how I think about inference optimization.
For large-batch cloud serving, generic runtimes and batching are often enough.
For realtime small-batch inference, the runtime overhead becomes the workload.
Curious if others have seen similar behavior with torch.compile, TensorRT, XLA, Triton, or custom CUDA kernels.
At what point do you stop trying to make a generic compiler optimize the model, and just rewrite the inference path directly?
Implementation: https://github.com/LiangSu8899/FlashRT
[link] [comments]
Read on the original site
Open the publisher's page for the full experience