Add Gemma 3n support (E2B, macOS) by stikves · Pull Request #198 · apple/coreai-models

stikves · 2026-08-26T01:37:50Z

Summary

Add Google Gemma 3n E2B for macOS (4-bit quantized)
Novel on-device architecture with AltUp, LAUREL, KV sharing, Gaussian TopK sparsity, dual RoPE, QKV norms
12 unit tests covering parity, KV sharing, sparsity, float16, AltUp, and tied embeddings

Evaluation

Compression	Wikitext PPL	tinyMMLU acc
none (float16)	61.53	58.75%
4-bit quantized	72.13	59.27%

High wikitext PPL is expected for instruction-tuned models — tinyMMLU accuracy is the appropriate quality metric.

Test plan

12 unit tests pass (parity, KV sharing, AltUp, sparsity, float16)
Export succeeds: uv run coreai.llm.export google/gemma-3n-E2B-it
Inference produces coherent text

stikves · 2026-08-29T02:36:46Z

(pausing this as the PR load is now high, will come back when others are resolved)

Add Google Gemma 3n E2B — a novel on-device architecture with AltUp (4-copy hidden state with predict/correct routing), LAUREL (low-rank residual), per-layer input embeddings, KV cache sharing, Gaussian TopK activation sparsity, dual RoPE, and QKV norms. The AltUp predict/correct operations use list-of-tensors (not a stacked 4D tensor) to avoid MPS Graph runtime crashes from strided views with dynamic sequence dimensions. KV-shared layers route cache reads through the mutable_cache_update_and_fetch custom op (zero-length write) for MPS compatibility. Evaluation (A100, bfloat16 compute, 4K context): | Compression | WikiText PPL | tinyMMLU | |-----------------|-------------|----------| | none (bfloat16) | 30.63 | — | | 4-bit quantized | 35.57 | 57.79% | Performance (M2 Max, 4-bit): 1042 t/s prefill, 50.8 t/s decode.

stikves force-pushed the sukru/gemma3n branch 3 times, most recently from c36cf23 to e389279 Compare August 26, 2026 01:54

stikves self-assigned this Aug 26, 2026

stikves force-pushed the sukru/gemma3n branch from e389279 to 2b3d111 Compare August 26, 2026 02:18