LLM Reddit 6d ago 2 min read
A LocalLLaMA thread highlighted ongoing work to add NVFP4 quantization support to llama.cpp GGUF, pointing to potential memory savings and higher throughput for compatible GPU setups.
#local-inference
LLM Reddit Mar 2, 2026 1 min read
A community developer achieved 100+ t/s decode speed and 585 t/s aggregate throughput for 8 simultaneous requests running Qwen3.5 27B on a dual RTX 3090 setup with NVLink, using vLLM with tensor parallelism and MTP optimization.
LLM Reddit Mar 2, 2026 1 min read
Alibaba released the Qwen3.5 small model series (0.8B, 4B, 9B). The 9B model achieves performance comparable to GPT-oss 20B–120B, making high-quality local inference accessible to users with modest GPU hardware.
AI Reddit Feb 21, 2026 2 min read
A high-upvote LocalLLaMA thread highlighted KittenTTS v0.8, with community-shared details on 80M/40M/14M model variants, Apache-2.0 licensing, and an edge-friendly focus on local CPU inference.
LLM Reddit Feb 20, 2026 2 min read
A popular LocalLLaMA post highlights draft PR #19726, where a contributor proposes porting IQ*_K quantization work from ik_llama.cpp into mainline llama.cpp with initial CPU backend support and early KLD checks.
LLM Reddit Feb 18, 2026 2 min read
A high-engagement LocalLLaMA post highlighted local deployment paths for MiniMax-M2.5, pointing to Unsloth GGUF packaging and renewed discussion on memory, cost, and agentic workloads.