LocalLLaMA Flags an Experimental Apple Neural Engine Backend for llama.cpp
Original: New - Apple Neural Engine (ANE) backend for llama.cpp View original →
A March 30, 2026 post in r/LocalLLaMA surfaced an experimental Apple Neural Engine backend for llama.cpp. By March 31, the thread had 68 points and 21 comments, enough to stand out because it points to a very specific attempt to move matrix work off the usual CPU and Metal path.
What is actually implemented
The Reddit post links to an issue comment in ggml-org/llama.cpp and the companion ggml-ane repository. In that comment, the author says the backend dispatches MUL_MAT operations to Apple’s Neural Engine through a private API. The same note describes it as a working ggml backend rather than an official upstream feature.
- The cited M4 Pro result is 4.0 TFLOPS peak at
N=256. - The author says that is 16.8x faster than CPU on the tested path.
- The prototype currently uses ANE for prefill at
N >= 64and falls back to Metal or CPU for decode.
The comment also mentions MIL-side transpose, a kernel cache, and support for quantized weights. Those details matter because they suggest the work is targeting real local-inference bottlenecks instead of merely proving that ANE can run a toy kernel. At the same time, the implementation relies on a private API, which is an important constraint for anyone expecting a production-ready or officially supported rollout.
Why it matters
Most Apple Silicon local-LLM stacks still split work between CPU and Metal, with the Neural Engine largely unused by open-source inference runtimes. If this experiment matures, it could create a third execution path for prefill-heavy workloads and reduce pressure on the GPU during mixed local workloads.
Even in its current state, the post is a useful signal: developers are testing whether ANE can become a serious inference target for ggml and llama.cpp, not just a hardware talking point. The community source is the Reddit thread; the primary technical source is the linked GitHub issue comment and prototype repository.
Related Articles
A recent r/LocalLLaMA benchmark thread argues that tokens-per-second screenshots hide the real trade-offs between MLX and llama.cpp on Apple Silicon. MLX still wins on short-context generation, but long-context workloads can erase that headline speedup because prefill dominates total latency.
A r/LocalLLaMA post pointed Mac users to llama.cpp pull request #20361, merged on March 11, 2026, adding a fused GDN recurrent Metal kernel. The PR shows around 12-36% throughput gains on Qwen 3.5 variants, while Reddit commenters noted the change is merged but can still trail MLX on some local benchmarks.
A rerun benchmark posted to r/LocalLLaMA argues that Apple’s M5 Max shows its clearest gains on prompt processing rather than raw generation alone. The post reports 2,845 tok/s PP512 for Qwen 3.5 35B-A3B MoE and 92.2 tok/s generation, but these remain community measurements rather than independent lab benchmarks.