LocalLLaMA Wants Qwen3.5-9B Quant Choices Backed by KLD, Not Vibes

The LocalLLaMA comparison for Qwen3.5-9B quantizations landed because it solved a very practical problem: there are too many GGUF files, and their names are not enough guidance. Instead of telling users to pick a popular upload, the post compares community quants against a BF16 baseline using mean KLD, or KL Divergence. In the author's framing, lower KLD means the quantized model's probability distribution stays closer to the original weights.

That metric choice is why the thread had technical weight. Perplexity can be noisy and dataset-sensitive; it can improve by accident on a test slice even when the model has drifted. KLD is not magic, but it directly asks how much the quantized distribution moved away from the baseline. For local users choosing between Q8_0, Q4 variants, i-quants, and provider-specific builds, that is a more useful starting point than file size alone.

The table highlighted near-lossless Q8-style options at the top, with multiple entries under a KLD score of 0.01. Commenters treated that as a shared reference rather than a final answer. Some asked for Gemma 4 and larger Qwen runs. Others suggested improving the chart with different marker shapes for different quant publishers. A longer technical comment praised the efficiency calculation while asking for KLD at near-full context lengths, because quantization can hurt long-context behavior even when short-context numbers look fine.

That is the community energy here: LocalLLaMA is moving from casual model recommendations toward repeatable measurement. The post does not decide one universal best quant. It gives users a way to talk about tradeoffs among file size, BPW, KLD, PPL, memory fit, and workload. For local inference, that is often the difference between chasing a filename and making an informed deployment choice.