I trained a 348M model trained from scratch on 22.7B tokens that does 14 digit arithmetic [P]
Our take
The sheer ingenuity of individual researchers pushing the boundaries of AI is consistently remarkable, and the recent work by u/nkthebass on a 348M parameter language model specializing in arithmetic is a compelling example. This project, detailed in a recent post, demonstrates impressive capabilities with a relatively small model size, trained on 22.7 billion tokens – a significant increase from their previous iteration. It's a testament to the power of focused training and architectural choices, particularly when compared to the massive scale often associated with state-of-the-art language models. The ability to achieve such high accuracy (99.4% average across nine GPT-3 arithmetic sub-tasks) with a model of this size challenges conventional wisdom about the necessity of enormous parameter counts for achieving strong performance in specific domains. This resonates with ongoing discussions about efficiency in AI, as highlighted in a recent piece on reducing image-processing token usage by nearly 95% compared to GPT-4o [I reduced image-processing token usage by ~95% compared with GPT-4o direct vision, while maintaining roughly the same accuracy.How significant is that?[P]].
What truly sets this model apart is its approach to solving arithmetic problems – by *showing its work*. Rather than simply outputting an answer, the model meticulously demonstrates the step-by-step process, mirroring human calculation methods like column addition and partial-product multiplication. This transparency is crucial for debugging and understanding the model's reasoning process, and it allows for a level of trust that is often lacking in black-box AI systems. The discovery that the model could invent new place values ("millions" and "ten-millions") when lacking vocabulary highlights an unexpected capacity for pattern recognition and extrapolation. The model’s limitations, particularly its struggles with word problems – achieving only 4% on the GSM8K benchmark – further underscore the complexities of natural language understanding and the need for specialized training approaches. As discussed in another recent article, Meta’s Muse Spark 1.3 boasts frontier performance, though its best results are currently tied to a model not broadly accessible [Meta says Muse Spark 1.3 has frontier performance — but its best results come from a model developers can’t broadly use yet]. This emphasizes the ongoing tension between achieving peak performance and democratizing access to advanced AI capabilities.
The detailed documentation, including the model card outlining failures and corrections based on community feedback, is a significant contribution to the open-source AI community. This level of transparency fosters collaboration and accelerates progress by allowing others to build upon and improve the model. The iterative process of training, with ten full fine-tuning rounds and the strategic use of LoRA adapters, demonstrates a thoughtful and methodical approach to model development. The observation that instruction tuning, while often beneficial, can sometimes *lower* benchmark scores on MC tasks is a valuable insight for practitioners and highlights the need for careful evaluation and optimization. The fact that someone independently tested the model and caught discrepancies, which were then promptly corrected, underscores the importance of rigorous validation and community involvement in ensuring the reliability of AI systems.
Ultimately, this project represents a significant step forward in the development of efficient and transparent AI models for specialized tasks. It demonstrates that achieving impressive performance doesn't always require massive scale and that a focused approach, combined with meticulous training and community feedback, can yield remarkable results. The question now is: how can we leverage these smaller, more interpretable models to build more reliable and trustworthy AI systems across a wider range of applications? The continued exploration of architectural innovations and training techniques, particularly those that prioritize transparency and explainability, will be crucial in shaping the future of AI.
Hello this is my fifth small language model I've made and apart of my third series and it has been a lot of work but it payed off: **348M parameters, 22.7B tokens**, then fine-tuned into a math model that solves arithmetic by *showing the work* — column addition with carries, borrow chains, partial-product multiplication — rather than guessing at an answer.
Last time I posted a 326M model trained on 10B tokens. This has about 2.3× the data, and the math side is WAY better than my previous two math models.
---
## The benchmarks
**99.4% average across the nine GPT-3 arithmetic sub-tasks**, which does much better past even where I trained it.
| Task | GPT-3 175B *(few-shot, direct)* | **This model (348M)** |
|---|:--:|:--:|
| 2-digit add | ~100% | **100%** |
| 3-digit add | 80.4% | **100%** |
| 4-digit add | 25.5% | **100%** |
| 5-digit add | 9.3% | **100%** |
| 2-digit sub | ~99% | **99.3%** |
| 3-digit sub | 94.2% | **98.3%** |
| 4-digit sub | 26.8% | **98.3%** |
| 5-digit sub | 9.9% | **99.0%** |
| 2-digit mult | 29.2% | **100%** |
n=300 per sub-task, greedy, exact match. GPT-3's numbers are direct-answer; mine uses trained-in worked steps. Neither uses a calculator.
## The cool part
**It adds cleanly up to 14 digits, and the reason it *couldn't* before was the vocabulary, not actually arithmetic.**
Training only ever named six place values (`ones` … `hundred-thousands`). The model learnt the *pattern* and invented two more on its own — `millions` and `ten-millions` appear in **zero** training examples — so it handled 7 and 8 digits fine. At 9 digits it ran out of names, and skipped the column, then returned an answer exactly one digit short:
```
483729164 + 519248637
... ten-millions: 8 + 1 + 1 (carry) = 10, write 0 carry 1. Final carry: write 1.
The answer is 102977801 ← eight columns for a nine-digit problem
```
Every column it computed was perfect. One was never enumerated. Extending the place-name list from 6 entries to 19 moved the clean ceiling from **8 digits to 14**:
| Width | 6 | 7 | 8 | 9 | 10 | 12 | 14 | 16 | 18 |
|---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| before | 100 | 100 | 100 | **0** | **0** | **0** | **0** | 0 | 0 |
| after | 100 | 100 | 100 | **100** | **95** | **100** | **90** | 65 | 25 |
A six-item list became a nineteen-item list. That was the entire fix.
## Other things it does
- **3×3 multiplication: 98%** — it folds partial products pairwise through the column routine instead of asserting the sum
- **Negative results: 85%** (100% at 1 digit, 58% at 5 — the magnitude comparison is the weak step, not the arithmetic)
- **Reasoning traces are load-bearing**: 95.3% of the time the working is valid *and* the answer is right; only 0.7% are "valid working, wrong answer." If the columns look right, the answer almost certainly is.
```
There were 15000 votes and 6842 were rejected. Here's how many counted:
<think> Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column:
ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1.
hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8,
borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158.</think>
The answer is 8158.
```
## What it's bad at, tbh
- **Word problems: GSM8K 4%.** Best word-problem set is ASDiv at 16.5%. It converts one sentence into one operation reasonably often and basically cannot chain operations.
- **The failure mode is operation *selection*, not arithmetic.** `"drops in 836 more"` gets read as subtraction. There's a visible tell: traces that say `"multiply X * Y"` and show columns are reliable; traces that open `"First, calculate…"` and assert a number in prose are not.
- **No division at all.** 4×4 multiplication is a hard wall.
- **Greedy decoding required** — sampling corrupts the column routine mid-chain.
- One caveat I'll flag myself: the arithmetic harness orders subtraction operands, so no answer in that table is negative. Negatives are measured separately (the 85% above) rather than folded into the average.
## Base and instruct
The math model sits on a base and instruct pair. lm-eval-harness, 0-shot, full test sets:
| | ARC-E | ARC-C | HellaSwag | OpenBookQA | PIQA | WinoGrande | MMLU | Avg |
|---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
| **350M V3 base** | 56.6 | 33.3 | 35.9 | 34.6 | 67.0 | 51.5 | 23.8 | **43.2** |
| **350M V3 instruct** | 50.9 | 29.7 | 35.9 | 33.8 | 66.6 | 51.2 | 24.4 | **41.8** |
Instruction tuning *lowers* MC benchmark scores for this family which is a pretty common cost of instruct tuning.
## Notes
Trained on 2× Tesla V100 plus some rented time. LLaMA-architecture, so it runs anywhere — F16 GGUF and safetensors for all three.
The math model took **10 full fine-tuning rounds and 3 LoRA adapters**. The LoRAs cost about 1% of the post-training tokens and did most of the useful work; the ten full rounds spent most of their budget undoing each other's regressions (round 7 gained subtraction and lost 23 points of 2-digit addition, that sort of thing). All of it is documented on the model card, failures included.
Also worth saying: someone independently tested it after I published and found two of my numbers were wrong — one *understated* the model by 40 points because I'd measured it at n=24. Both are corrected on the card now. If you find something broken, I'd genuinely like to know.
**Math:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-math
**Instruct:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-instruct
**Base:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-base
LMK what yall think.
[link] [comments]
Read on the original site
Open the publisher's page for the full experience