LocalLLaMA digs into Gemma 4 Per-Layer Embeddings and why the small models behave differently

A technical LocalLLaMA post offered a useful mental model for understanding why the smaller Gemma 4 models look unusual on paper. The author argues that gemma-4-E2B and gemma-4-E4B should not be read as standard dense models or as classic Mixture-of-Experts systems. Instead, the important feature is Per-Layer Embeddings, or PLE, which appear to shift where the parameter count sits and how much of it needs to be actively involved in each inference step.

The post contrasts this with Gemma 4's MoE variant, gemma-4-26B-A4B. In an MoE model, only a subset of experts is active per token, but the full weight set still has to be available in fast memory because the router may pick different experts from token to token. For gemma-4-E2B, the author cites 5.1B total parameters, with 2.8B attributed to embedding parameters and roughly 2.3B treated as “effective” parameters. The explanation offered is that embeddings are better understood in serving systems as lookup tables than as giant matrix multiplies applied wholesale every time.

Why that matters for inference

The practical implication is that PLE-heavy models may carry a large parameter count without forcing the same compute and memory behavior as a comparably sized dense model. If only the entries tied to the tokens actually present in the request need to be fetched, then a large share of the model can behave more like selectively accessed data than continuously active arithmetic. That is the intuition behind the post's claim that these embedding-heavy weights do not “count” in the same way for runtime performance.

This framing also helps explain why developers are interested in Gemma 4 for on-device use. The post argues that the relevant embedding data does not necessarily need to live in VRAM at all times and could, in principle, be handled through RAM or storage-backed access patterns, depending on the implementation. That does not replace official documentation, but it does offer a clearer reason for why the Gemma 4 E-series naming exists and why the small models can look surprisingly capable for their advertised effective size.

A community explainer with real value

It is worth keeping the source in perspective: this is a community explainer, not a formal Google architecture paper. Exact implementation details should still be checked against model cards and future technical notes. Even so, the post is valuable because it gives developers a more precise vocabulary for discussing Gemma 4 than the usual dense-versus-MoE shorthand. For anyone evaluating inference paths on laptops, phones, or other edge devices, that is a useful correction.