Colibri Runs GLM-5.2 on a Slow PC, and the Real Debate Is Memory Movement

Colibri is a small-engine experiment for running GLM-5.2, a 744B-parameter Mixture-of-Experts model, on consumer hardware with roughly 25GB of RAM. The design leans on the fact that a sparse MoE does not activate every parameter for every token. Dense parts stay resident in RAM, while routed experts live on disk and are streamed as needed.

The interesting part is not the slogan of “huge model on a slow computer,” but the engineering boundary it chooses. The project describes a pure C runtime with no Python, BLAS, or GPU requirement, plus validation against a transformers oracle. Its README says the dense section is kept in int4 form, while tens of thousands of routed experts are handled through a per-layer LRU cache, an optional pinned hot store, and the operating system page cache.

That framing is why the HN thread quickly moved from novelty to throughput. Commenters asked whether practical use means tokens per second or something closer to one token per minute. Others compared the approach with mmap behavior in llama.cpp, aggressive quantization, and unified memory on Apple Silicon. The real question is not whether a large MoE can be made to run; it is which workloads remain useful when storage and memory movement dominate.

Colibri reads less like a production inference stack and more like a useful stress test for local AI. Instead of shrinking the model until it fits, it asks whether the active slice can be read cheaply enough. If that path keeps improving, the next local-inference gains may come from storage layout, cache policy, compressed KV-cache design, and smarter routing rather than raw GPU size alone.

HN discussion · Source