LLM Reddit May 12, 2026 1 min read
A LocalLLaMA user built a 768GB RAM system using discontinued Intel Optane Persistent Memory from the secondhand market, running the 1-trillion-parameter Kimi K2.5 model locally at over 4 tokens per second.
#llm-inference
RSS FeedAI X/Twitter Apr 18, 2026 2 min read
Why it matters: Cloudflare is attacking the memory-bandwidth bottleneck in LLM serving rather than only buying more GPUs. Its post reports 15-22% model-size reduction, about 3 GB VRAM saved on Llama 3.1 8B, and open-sourced GPU kernels.
LLM Reddit Mar 29, 2026 3 min read
A March 2026 r/singularity post shared Google Research’s TurboQuant work and drew 114 points with 18 comments. Google says the method can shrink KV cache memory by at least 6x on needle tasks, quantize caches to 3 bits without training, and deliver up to 8x attention-logit speedups on H100 GPUs.
LLM Reddit Mar 29, 2026 2 min read
A high-scoring r/LocalLLaMA post explains TurboQuant not as a polar-coordinates trick but as random rotation before quantization. The linked arXiv paper claims near-optimal distortion rates, a residual QJL stage for inner products, and quality-neutral KV cache quantization at 3.5 bits per channel.
LLM Reddit Mar 28, 2026 2 min read
A popular r/LocalLLaMA post revived attention around Google Research’s TurboQuant by tying it directly to local inference constraints. The method’s reported 3-bit KV cache compression and 6x memory reduction make it relevant well beyond research headlines, but its practical value will depend on whether it reaches real deployment stacks.
LLM Reddit Mar 27, 2026 2 min read
A LocalLLaMA self-post shared an open-source TurboQuant implementation for llama.cpp that skips value dequantization when attention weights are negligible. The author reports a 22.8% decode gain at 32K context on Qwen3.5-35B-A3B over Apple M5 Max, with unchanged perplexity and better needle-in-a-haystack retrieval.
LLM Reddit Mar 27, 2026 2 min read
The Reddit thread focused on a practical claim with real systems implications: replace TurboQuant's dense rotation with structured rotor math, keep attention fidelity close, and make the kernel much cheaper on NVIDIA and Apple hardware.
LLM Hacker News Mar 25, 2026 2 min read
Hacker News noticed Hypura because it treats Apple Silicon memory limits as a scheduling problem, spreading tensors across GPU, RAM, and NVMe instead of letting oversized models crash.
LLM Hacker News Mar 13, 2026 2 min read
CanIRun.ai runs entirely in the browser, detects GPU, CPU, and RAM through WebGL, WebGPU, and navigator APIs, and estimates which quantized models fit your machine. HN readers liked the idea but immediately pushed on missing hardware entries, calibration, and reverse-lookup features.
AI Reddit Mar 1, 2026 1 min read
A developer has implemented a UEFI application that runs LLM inference directly from boot without any operating system or kernel, using zero-dependency C code for the entire stack from tokenizer to inference engine.
LLM Reddit Feb 26, 2026 1 min read
A trending r/LocalLLaMA thread highlighted the DualPath paper on KV-Cache bottlenecks in disaggregated inference systems. The arXiv abstract reports up to 1.87x offline throughput and 1.96x average online throughput gains while meeting SLO.
LLM Hacker News Feb 21, 2026 2 min read
A high-score Hacker News discussion surfaced Together AI's CDLM post, which claims up to 14.5x latency improvements for diffusion language models by combining trajectory-consistent step reduction with exact block-wise KV caching.