Tiny-vLLM teaches LLM inference by rebuilding the stack in C++ and CUDA
Original: Show HN: Tiny-vLLM – high performance LLM inference engine in C++ and CUDA View original →
Tiny-vLLM is a compact LLM inference engine written in C++ and CUDA. It loads a real Llama 3.2 1B Instruct model from Safetensors, runs prefill and decode, implements CUDA kernels, KV cache, static and continuous batching, online softmax, FlashAttention-style ideas and PagedAttention. The repository is also structured as a course, walking readers through each layer of the system.
That combination explains the Hacker News attention. Production inference frameworks are powerful, but they are not easy first texts. Tiny-vLLM takes the opposite route: start with the model file, explain weights and tensor shapes, move through tokenization, embeddings, RMSNorm, RoPE, attention, CUDA kernels, batching and paged KV cache. It is not trying to hide the machinery behind a clean API. It is trying to make the machinery small enough to inspect.
The community discussion repeatedly praised the README. The author said the documentation was written to help others build a mental model without reading every line of code first, and commenters called out the lesson format as unusually approachable. One thread compared it to the early energy around llama.cpp, where a small codebase made a complex system feel reachable.
The scope is still intentionally narrow. Tiny-vLLM targets a specific model family and NVIDIA CUDA setup, and it is not a replacement for full serving stacks with scheduling, observability, isolation and operational controls. Its value is educational and architectural. For developers trying to understand why KV cache exists, how batching changes throughput, or what PagedAttention buys in memory behavior, a small implementation can be more useful than another high-level overview.
Related Articles
The Reddit thread is not about mourning TGI. It reads like operators comparing notes after active momentum shifted away from it, with most commenters saying vLLM is now the safer default for general inference serving because the migration path is lighter and the performance case is easier to defend.
The community interest came from a practical question: can a huge MoE model be useful on ordinary hardware? Colibri uses GLM-5.2’s sparse activation pattern to avoid loading the whole model into RAM or a GPU at once.
A March 26, 2026 r/LocalLLaMA post about serving Qwen 3.5 27B on Google Cloud B200 clusters reached 205 points and 52 comments at crawl time. The linked write-up reports 1,103,941 total tokens per second on 12 nodes after switching from tensor to data parallelism, shrinking context length, enabling FP8 KV cache, and using MTP-1 speculative decoding.