Gemma 4 26B runs at 5 tok/s on a 13-year-old Xeon
Original: Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU View original →
A 13-year-old Xeon box running Gemma 4 26B at about five tokens per second is not a story about nostalgia. It is a useful boundary test for local inference. The hardware is a repurposed HP StoreVirtual system with dual Xeon E5-2690 v2 CPUs, DDR3 memory, no GPU, AVX1 support, and no AVX2 or FMA3. That is well below the assumptions many modern inference paths quietly make.
The interesting part is the failure mode. The author used ikawrakow’s ik_llama.cpp fork, which contains optimizations needed for Gemma 4’s MoE inference. On pre-AVX2 hardware, the build and runtime paths did not line up cleanly. Some fused operations were still emitted by the graph builder even though the non-AVX2 dispatcher had no working case for them. The result was not a crash or NaNs. It was deterministic multilingual nonsense caused by tensors that never got computed.
The fix decomposed the AVX2-only fused up-gate path into operations that already worked on the old CPU: separate matmul calls plus a SILU-and-multiply step. The branch also included scalar compile fixes and CI stub cleanup so the fork could build and run on non-AVX2 hardware. With the patch, Gemma 4 26B-A4B in Q8_0 reaches roughly 5.2 tok/s decode and 16 tok/s prompt evaluation, CPU-only.
HN discussion quickly turned to economics. Some commenters argued that electricity and cooling make provider inference cheaper per token, especially at five tokens per second. Others focused on control: if the hardware is already sitting around, a local model can be useful for private workloads, slow batch jobs, offline use, or fallback capacity when paid APIs are unavailable. The point is not that old servers beat hosted inference in every case. It is that the tradeoff is specific enough to calculate.
The debugging story also matters. This was not a single prompt producing a magic patch. A human had to run experiments, inspect logs, know what correct output should look like, and keep narrowing the failure. The agent helped dig through unfamiliar performance-sensitive C++ once the problem was framed tightly. That may be the more durable lesson: local AI is not only about buying newer GPUs. Sometimes it is about making old hardware useful by understanding exactly where the software assumed it was newer.
Related Articles
The popular thread turned a local-inference stunt into a practical discussion about decoding bottlenecks, power cost, and runtime knobs.
The community interest came from a practical question: can a huge MoE model be useful on ordinary hardware? Colibri uses GLM-5.2’s sparse activation pattern to avoid loading the whole model into RAM or a GPU at once.
A March 17, 2026 r/LocalLLaMA post about Hugging Face hf-agents reached 624 points and 78 comments at crawl time. The extension uses llmfit to detect hardware, recommends a runnable model and quant, starts llama.cpp, and launches the Pi coding agent.