CUDA Graphs
CUDA Graphs on Beyond Market Intelligence: a running collection of 2 stories we have gathered and hand-picked because they are worth your time. Every post here touches on cuda graphs in some way — the news, the analysis, the deep dives, and the occasional surprise find. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work with data. New stories are added to this page as we find them, so check back if you want to keep up with what is happening around cuda graphs, or subscribe to the RSS feed to get them as soon as they are published. Browse the collection below, or head back to the homepage to see everything Beyond Market Intelligence is covering right now.
28 TPS on Qwen2.5-7B across two separate cloud regions over public WAN using speculative decoding + CUDA Graphs [P]
Recent benchmarking demonstrates transformative gains in distributed LLM inference using ShardFlow, a new framework designed to split HuggingFace transformers across multiple GPUs. Utilizing speculative decoding and CUDA Graphs, we achieved a peak of 28.10 tokens per second (TPS) on Qwen2.5-7B across two geographically separated GCP regions (Iowa & Oregon) connected via a public WAN – a setup exhibiting approximately 86ms round-trip latency. This highlights a key insight: speculative decoding minimizes latency impact, shifting costs from per-token to per-round.
How To Build Your Own LLM Runtime From Scratch
Ever wondered what it takes to build an LLM inference runtime from the ground up? This comprehensive guide details that journey, walking you through the creation of a small runtime called annotated-llm-runtime, all while running on an H100. We explore the intricacies of managing weights and CUDA graphs, highlighting three key bugs that shaped the development process. Delve into the complexities of AI infrastructure—as explored further in "OpenAI’s AI spending spree has ballooned to $750B"—and empower yourself with a deeper understanding of LLM technology.