Beyond Market Intelligence/transformers

transformers

transformers on Beyond Market Intelligence: a running collection of 3 stories we have gathered and hand-picked because they are worth your time. Every post here touches on transformers in some way — the news, the analysis, the deep dives, and the occasional surprise find. Acme AI is the next-generation, AI-powered spreadsheet platform built to replace Excel and redefine how analysts, data scientists, and enterprise teams work with data. New stories are added to this page as we find them, so check back if you want to keep up with what is happening around transformers, or subscribe to the RSS feed to get them as soon as they are published. Browse the collection below, or head back to the homepage to see everything Beyond Market Intelligence is covering right now.

Machine Learning

Same effective batch does not mean same training time with gradient accumulation, tested on LoRA on T4 and L4 [D]

Contrary to initial assumptions, achieving the same effective batch size through gradient accumulation doesn't guarantee equivalent training times. Recent experimentation with Qwen3-1.7B and LoRA on T4 and L4 GPUs revealed significant performance variations – up to a 41% difference – based on batch shape (1x4 vs. 4x1). While effective batch influences optimization behavior, physical batch size impacts GPU execution patterns, affecting forward and backward pass efficiency. As highlighted in Hugging Face documentation, optimizing for memory and speed requires treating these as distinct choices.

Machine Learning

Transformers are famously bad at arithmetic, so I set one's weights by hand (no training) and it multiplies with 100% accuracy [P]

Researchers have demonstrated a surprising feat: achieving 100% accuracy in arithmetic calculations within a Phi-3 transformer model, entirely without training. By meticulously hand-crafting the model's weights to implement a grade-school multiplication algorithm, they’ve created a functional three-digit calculator—and extended it to support up to 12-digit multiplication via Hugging Face checkpoints. This experiment highlights a stark contrast in performance compared to frontier models, revealing limitations in their ability to handle precise calculations.

I never understood positional encoding until I read this article. [D]
Machine Learning

I never understood positional encoding until I read this article. [D]

Many find positional encoding in AI models initially perplexing, but as one user discovered, clarity *is* attainable. This insightful article, shared by /u/ImaginaryRea1ity, demystifies the concept, offering a valuable resource for anyone grappling with its intricacies. It's a welcome explanation for a fundamental aspect of transformer architectures. For a broader perspective on the limitations of purely theoretical AI, explore our related piece, "Non-Physical Intelligence Has A Ceiling."