I compiled Doom's renderer into a 21B-parameter transformer -- no training anywhere [P]

The recent demonstration of rendering the Doom engine within a transformer model, as detailed in the I Made an LLM Lay Siege to My Minecraft House post, represents a fascinating confluence of established and emerging AI technologies. It moves beyond the typical LLM applications of text generation and summarization, showcasing a novel approach to computation and representation. Instead of training a model to learn the rendering process, the author, physicsrob, has cleverly compiled the existing Doom rendering algorithm—a meticulously crafted piece of code from the 1990s—into a transformer architecture. This avoids the enormous computational cost of training, relying instead on a custom compiler that translates the algorithm’s computational graph into transformer weights. The result is a functional, albeit slow, Doom renderer operating within a Hugging Face transformer, a development that highlights the potential for repurposing existing codebases within AI frameworks. This builds upon the foundational explorations of how AI is changing workflows, as illustrated in A Day in the Life of a Data Scientist in 2026 where AI tools are fundamentally reshaping the data science landscape.

The technical ingenuity of this project lies not just in the compilation process, but also in the chosen output format. Instead of generating pixels directly, the transformer outputs a sequence of drawing commands – move cursor, draw pixel, etc. This approach, while unconventional, demonstrates a surprising degree of flexibility within the transformer architecture. The fact that this can be achieved without any training whatsoever is particularly noteworthy, further emphasizing the potential of compiler-based approaches to AI. The contrast between the original Doom’s impressive 35 FPS on 1990s hardware and the 35 Frames Per Day achieved on a B200 underscores the current computational limitations, but the sheer novelty of the concept outweighs these performance concerns. The project’s accessibility, with readily available weights and source code, encourages further exploration and experimentation within the community, furthering the innovative use cases of LLMs beyond their intended scope.

The broader significance of this work extends beyond a nostalgic recreation of a classic game. It suggests a potential paradigm shift in how we approach complex software execution within AI systems. Traditional AI often relies on training models to perform specific tasks, but this project demonstrates the possibility of repurposing existing, well-understood algorithms by translating them into a compatible format. This could open up avenues for integrating legacy codebases, scientific simulations, and other computationally intensive applications into AI workflows, potentially accelerating innovation across various fields. The underlying technology, the "torchwright" compiler, promises a future where existing software isn't replaced by AI, but rather, *integrated* into it. It's an interesting parallel to how we’re seeing AI applied to web scraping, as demonstrated in How to Build a Simple AI Web Scraper with Python, transforming data extraction into a more streamlined process.

Looking ahead, a key question is whether this approach can be scaled to more complex and computationally demanding applications. While rendering Doom is a compelling demonstration, the current performance limitations highlight the challenges of translating complex algorithms into transformer-compatible formats. Future research will likely focus on optimizing the compilation process, exploring alternative transformer architectures, and potentially combining compiler-based approaches with targeted training to achieve both efficiency and accuracy. The ability to effectively “port” existing software into AI systems, bypassing the traditional training bottleneck, could unlock a new era of AI-powered applications, moving us closer to a future where AI seamlessly integrates with the world around us.