#quantization

Hacker News pushed Microsoft's bitnet.cpp back into view, treating it less as a new 100B checkpoint and more as an infrastructure play for 1.58-bit inference and lower-power local LLM deployment.

A LocalLLaMA thread highlighted ongoing work to add NVFP4 quantization support to llama.cpp GGUF, pointing to potential memory savings and higher throughput for compatible GPU setups.

A high-scoring LocalLLaMA post benchmarked Qwen3.5-27B Q4 GGUF variants against BF16, separating “closest-to-baseline” choices from “best efficiency” picks for constrained VRAM setups.

A high-engagement r/LocalLLaMA thread reviewed Unsloth’s updated Qwen3.5-35B-A3B dynamic quantization release, including KLD/PPL data, tensor-level tradeoffs, and reproducibility artifacts.

A high-engagement LocalLLaMA follow-up benchmark reports that Qwen3.5-35B-A3B runs best on the tested RTX 5080 setup with Q4_K_M quantization, KV q8_0, and --fit without explicit batch flags.

A popular LocalLLaMA post highlights draft PR #19726, where a contributor proposes porting IQ*_K quantization work from ik_llama.cpp into mainline llama.cpp with initial CPU backend support and early KLD checks.

A high-engagement LocalLLaMA post highlighted local deployment paths for MiniMax-M2.5, pointing to Unsloth GGUF packaging and renewed discussion on memory, cost, and agentic workloads.

A r/MachineLearning discussion reported that one INT8 ONNX model produced large on-device accuracy variance across five Snapdragon chipsets, from 91.8% down to 71.2%, despite identical weights and export settings.

A popular r/LocalLLaMA post details Heretic 1.2 with PEFT/LoRA updates, optional 4-bit processing, MPOA support, VL coverage, and automatic resume features for long local optimization runs.