Tiny Transformers (<100 Params) Add Two 10-Digit Numbers with 100% Accuracy

Tiny Models, Perfect Arithmetic

A striking finding has earned 138 upvotes on r/MachineLearning: transformer models with fewer than 100 parameters can add two 10-digit numbers with 100% accuracy. The results are published in the AdderBoard GitHub project, and they have implications beyond just arithmetic.

The Key: Digit Tokenization

The critical insight is in how numbers are tokenized. When numbers are represented as individual digit tokens rather than as floating-point values or opaque number strings, the model can learn place-value addition directly. Community commentary notes that floating-point math would be far trickier — but digit tokens make the problem tractable even for extremely small models.

Implications for LLM Mathematical Reasoning

This research raises an interesting question: why do large language models often struggle with multi-digit arithmetic when tiny transformers can do it perfectly? One key reason is that standard LLM tokenizers often bundle multiple digits into a single token, obscuring the underlying place-value structure that makes addition learnable.

The findings suggest that digit-aware tokenization could be a meaningful component of specialized math-capable models. More broadly, the result illuminates the relationship between tokenization choices and emergent mathematical capabilities — a question increasingly relevant as the field pushes LLMs into more rigorous reasoning domains.