A 28.9M-parameter LLM runs on an $8 ESP32, with flash doing the heavy lifting

A 28.9M-parameter language model is running directly on an ESP32-S3 microcontroller. The GitHub project describes an $8 chip with 512KB SRAM, 8MB PSRAM, and 16MB flash generating text locally at roughly 9 tokens per second, with nothing sent to a server. The author frames it as roughly two orders of magnitude larger than the previous language model run on a similar chip.

The core trick is memory placement. Instead of trying to keep the whole model in fast memory, the project stores most parameters in flash. A 25M-parameter lookup table stays there, while each token reads only the small number of rows it needs. The fast SRAM holds the small compute core used on every token, and PSRAM handles the output head and working memory. The idea draws on Per-Layer Embeddings from Google’s Gemma models, applied to a much smaller hardware envelope.

The README is careful about the limit. The model was trained on TinyStories, so it writes short, simple stories and mostly keeps them coherent. It is not a question-answering system, coding model, or factual assistant. That actually makes the experiment cleaner: the point is not to pretend an ESP32 has a frontier model inside it, but to show a practical way to fit a surprisingly large language model onto a tiny device.

The HN thread reacted to the hardware implications. Commenters connected the work to small offline speech devices, local TTS-scale models, and newer low-cost boards with more memory or accelerators. Others focused on the Per-Layer Embedding approach itself. For edge AI, this is the useful lesson: model size is no longer just a parameter count, but a question of which weights must be fast, which can be slow, and how little data each token actually touches.

Sources: GitHub repository and HN discussion.