Swiftlet streams Qwen experts instead of fitting the whole model in RAM

Swiftlet is a Swift and Metal runtime for running large Qwen MoE models on Apple devices without keeping the whole model resident in memory. The project’s headline numbers are deliberately concrete: Qwen3.6-35B-A3B at 4-bit uses an 18 GB container and about 2.6 GB peak RAM, while Qwen3-Next-80B-A3B uses 42 GB on disk and about 4.3 GB peak RAM. On an M5 Mac, the README reports roughly 7 to 11 tokens per second for the 35B model and 4.5 to 5 tokens per second for the 80B model. The 35B model also runs on an iPhone 17 at about 2.5 GB RAM and roughly 1 token per second today.

The design hinges on sparse activation. Swiftlet keeps the dense core in memory: attention, DeltaNet projections, routers, shared experts, embeddings, and related always-needed weights. Routed Mixture-of-Experts weights are repacked into a .qpack container and read from storage on demand. Each expert is placed in a fixed-stride blob, so fetching one expert becomes a single pread rather than a page-cache-heavy mmap pattern. A bounded cache uses frequency and recency to retain hot experts.

That distinction matters because the project is not claiming that an iPhone suddenly behaves like a datacenter GPU. The README says only about 3B parameters are active per token, so these models can chat and write like larger models while recalling facts more like smaller ones. It also notes that current decode speed is dispatch-bound rather than I/O-bound, leaving optimization headroom in Metal kernels and scheduling.

The HN discussion treated the project as an on-device inference experiment with real tradeoffs. Commenters asked about SSD wear, tunable RAM caches for larger Macs, and whether consumer devices will eventually run everyday LLM workloads locally. The original source is the Swiftlet repository, with community discussion on Hacker News.