The HN debate was not just “old hardware still works.” A patched ik_llama.cpp path got Gemma 4 26B-A4B running CPU-only on dual Ivy Bridge Xeons, raising practical questions about local inference cost, control, and fallback capacity.
A community user achieved 110 tokens/second running Qwen3.6 35B A3B on an RTX 4070 Super 12GB via ik_llama.cpp, a fork with superior CPU offload optimization that significantly outperforms upstream llama.cpp's Multi-Token Prediction implementation.
A LocalLLaMA user built a 768GB RAM system using discontinued Intel Optane Persistent Memory from the secondhand market, running the 1-trillion-parameter Kimi K2.5 model locally at over 4 tokens per second.
A LocalLLaMA user shares their config for running Qwen3.6 35B A3B at over 80 tok/sec with 128K context on a 12GB VRAM GPU, using llama.cpp's Multi-Token Prediction support and achieving 80%+ draft acceptance rate.
llama.cpp's Multi-Token Prediction (MTP) support has entered beta, currently covering Qwen3.5 MTP. Combined with maturing tensor-parallel support, most token generation speed gaps between llama.cpp and vLLM are expected to close.
The community liked this post for the same reason it immediately started arguing with it: it had real numbers. Q4_K_M came out looking like the practical sweet spot, but commenters quickly pushed on error bars, KV-cache settings, and whether the reported scores made sense at all.
LocalLLaMA latched onto a very concrete claim: if a 27B model fits entirely in VRAM across two mismatched cards, even a weak second GPU can be better than spilling into system RAM for long-context decoding.
LocalLLaMA reacted because this was not just a translation app; it chained detection, visual OCR, inpainting, and local LLM choices into one workflow.
LocalLLaMA reacted because --fit challenged the old rule of thumb that anything outside VRAM means painfully slow inference.
r/LocalLLaMA cared because the numbers were concrete: 79 t/s on an RTX 5070 Ti with 128K context, tied to one llama.cpp flag choice.
LocalLLaMA reacted because the post attacks a very real pain point for running large MoE models on limited VRAM. The author tested a llama.cpp fork that tracks recently routed experts and keeps the hot ones in VRAM for Qwen3.5-122B-A10B, reporting 26.8% faster token generation than layer-based offload at a similar 22GB VRAM budget.
LocalLLaMA reacted because the joke-like idea of an LLM tuning its own runtime came with concrete benchmark numbers. The author says llm-server v2 adds --ai-tune, feeding llama-server help into a tuning loop that searches flag combinations and caches the fastest config; on their rig, Qwen3.5-27B Q4_K_M moved from 18.5 tok/s to 40.05 tok/s.