Tiny-vLLM teaches LLM inference by rebuilding the stack in C++ and CUDA

Tiny-vLLM is a compact LLM inference engine written in C++ and CUDA. It loads a real Llama 3.2 1B Instruct model from Safetensors, runs prefill and decode, implements CUDA kernels, KV cache, static and continuous batching, online softmax, FlashAttention-style ideas and PagedAttention. The repository is also structured as a course, walking readers through each layer of the system.

That combination explains the Hacker News attention. Production inference frameworks are powerful, but they are not easy first texts. Tiny-vLLM takes the opposite route: start with the model file, explain weights and tensor shapes, move through tokenization, embeddings, RMSNorm, RoPE, attention, CUDA kernels, batching and paged KV cache. It is not trying to hide the machinery behind a clean API. It is trying to make the machinery small enough to inspect.

The community discussion repeatedly praised the README. The author said the documentation was written to help others build a mental model without reading every line of code first, and commenters called out the lesson format as unusually approachable. One thread compared it to the early energy around llama.cpp, where a small codebase made a complex system feel reachable.

The scope is still intentionally narrow. Tiny-vLLM targets a specific model family and NVIDIA CUDA setup, and it is not a replacement for full serving stacks with scheduling, observability, isolation and operational controls. Its value is educational and architectural. For developers trying to understand why KV cache exists, how batching changes throughput, or what PagedAttention buys in memory behavior, a small implementation can be more useful than another high-level overview.