Transformers are famously bad at arithmetic, so I set one's weights by hand (no training) and it multiplies with 100% accuracy [P]

The recent demonstration of a transformer model performing accurate multiplication, achieved not through training but through direct weight manipulation, is a fascinating development highlighting the untapped potential within existing AI architectures. It’s a stark reminder that even models seemingly ill-suited for specific tasks can be coaxed into performing them with surprising precision when approached creatively. This work builds on the broader exploration of smaller, more efficient language models, a trend we’ve seen gaining traction, as evidenced by articles like Small Language Models with Hugging Face transformers Library + smolLM3 which explores the viability of running powerful models in resource-constrained environments. Furthermore, the focus on accessible AI tools, as showcased in 5 Free Courses to Learn Modern AI and LLMs, reinforces the idea that powerful AI capabilities don’t always necessitate massive infrastructure or specialized expertise. The project's success, using a custom compiler called Torchwright, underlines the importance of tooling and infrastructure in unlocking new possibilities within established frameworks.

The core innovation here isn't necessarily the multiplication itself – as the author playfully notes, nobody *needs* a multiplication-specific transformer. Rather, it’s the methodology: embedding a proven algorithm (the grade-school multiplication method) directly into the weights of a transformer, effectively bypassing the traditional training process. This approach bypasses the inherent limitations of training transformers, which are primarily designed for probabilistic language modeling, not deterministic computation. The comparison with frontier models, which falter dramatically with increasing digit lengths, further underscores the elegance and efficiency of this weight-based solution. By contrasting different implementations – grade-school, hardware-style, scratchpad, and brute-force – the author highlights the diverse ways in which the same function can be computed, showcasing the flexibility inherent in transformer architectures. This is a significant departure from the prevailing paradigm of large-scale pre-training and fine-tuning.

The implications extend beyond mere computational feats. This work suggests a new avenue for exploring the capabilities of existing transformer models – a shift from focusing solely on training new models to creatively repurposing and configuring existing ones. It opens doors to embedding specialized algorithms and knowledge directly into model weights, potentially leading to more efficient and targeted AI solutions. The availability of the checkpoints on Hugging Face, along with the open-source Torchwright compiler, is particularly valuable, enabling others to experiment with this approach and potentially adapt it to other computational tasks. The accessibility of solutions like those presented in No cloud, no GPUs, no problem: Liquid AI's new model LFM2.5-2.6B brings powerful AI agents to devices as small as a Raspberry Pi demonstrates the growing trend toward democratizing AI, and this project fits comfortably within that narrative.

Ultimately, this experiment challenges our assumptions about the limitations of transformer models and suggests a future where AI development involves more than just scaling up models. It encourages a more nuanced understanding of how to leverage existing architectures to solve specific problems, potentially unlocking unforeseen capabilities without the immense computational resources typically required. The success of this project raises a compelling question: what other deterministic algorithms, currently outside the scope of typical transformer applications, could be embedded in a similar fashion, and what new efficiencies and capabilities might that unlock?