#inference

A Launch HN thread pushed RunAnywhere's RCLI into view as an Apple Silicon-first macOS voice AI stack that combines STT, LLM, TTS, local RAG, and 38 system actions without relying on cloud APIs.

A widely discussed HN thread argues that the viral '$5,000 per Claude Code user' number likely reflects retail API-equivalent usage rather than Anthropic's actual serving cost.

A LocalLLaMA thread reported a large prompt-processing speedup on Qwen3.5-27B by lowering llama.cpp `--ubatch-size` to 64 on an RX 9070 XT. The interesting part is not a universal magic number, but the reminder that prompt ingestion and token generation can respond very differently to `n_ubatch` tuning.

A r/LocalLLaMA thread is drawing attention to `llama.cpp` pull request #19504, which adds a `GATED_DELTA_NET` op for Qwen3Next-style models. Reddit users reported better token-generation speed after updating, while the PR itself includes early CPU/CUDA benchmark data.

A well-received PSA on r/LocalLLaMA argues that convenience layers such as Ollama and LM Studio can change model behavior enough to distort evaluation. The more durable lesson from the thread is reproducibility: hold templates, stop tokens, sampling, runtime versions, and quantization constant before judging a model.

Microsoft Research presented new tiny language model (TLM) results focused on reasoning efficiency at edge scale. The post emphasizes bitnet-based small models, 2-bit ternary weights, and reported gains of up to 8x speed with 4x lower memory in selected environments.

NVIDIA AI Developer says a collaboration with SGLang achieved up to 25x faster DeepSeek R1 inference on GB300 NVL72 versus H200 and an 8x GB200 NVL72 gain within months. The post attributes gains to NVFP4 precision, disaggregation, and communication-compute overlap.

A Show HN project called Timber claims it can compile tree-based ML models into dependency-free C99 artifacts, with reported ~2 microsecond latency and up to 336x speedup over Python baselines.

Andrej Karpathy highlights the fundamental memory+compute trade-off challenge in LLMs: fast but small on-chip SRAM versus large but slow off-chip DRAM. He calls optimizing this the most intellectually rewarding puzzle in AI infrastructure today, pointing to NVIDIA's $4.6T market cap as proof.

Startup Taalas proposes baking entire LLM weights and architecture into custom ASICs, claiming 17K+ tokens/second per user, sub-1ms latency, and 20x lower cost than cloud — all achievable within a 60-day chip production cycle.

Taalas has released an ASIC chip that physically etches Llama 3.1 8B model weights into silicon, achieving 17,000 tokens per second—10x faster, 10x cheaper, and 10x more power-efficient than GPU-based inference systems.

A new open-source project called ntransformer enables running the 140GB Llama 3.1 70B model on a single consumer RTX 3090 by streaming weights directly from NVMe storage to GPU, completely bypassing CPU RAM.