Running a 14.4 GB Gemma 4 26B MoE on an 8 GB RTX 3050 with llama.cpp

What they built

This is part five of a series on running Gemma 4 on a budget card. The 26B A4B mixture-of-experts model file is 14.4 GB, far bigger than the 8 GB RTX 3050, but 89% of it is expert weights and each token only reads 8 of 128 experts per layer.

llama.cpp's -ncmoe option keeps the experts of the first N layers in system RAM, where the CPU runs them, and puts everything else on the GPU. The author measured how fast it generates, how much context fits, what Google's QAT drafter adds, and whether answers match the Gemma 4 12B that fits on the card whole.

The repo also includes a suite of Python MCP tools for managing the llama.cpp deployment.

Setup

• RTX 3050 OEM (8 GB) in an i7-5820K desktop with 6 cores, quad-channel DDR4 and 46 GiB RAM
• llama.cpp build 7182ef71e, one request at a time (--parallel 1)
• Google's google/gemma-4-26B-A4B-it-qat-q4_0-gguf, unmodified
• An exact Q4_0 QAT drafter (252 MB) for speculative decoding, 4-bit context cache

How it works

1. -ngl 99 puts every layer on the GPU, then -ncmoe 21 moves the experts of the first 21 layers back to system RAM.
2. Each expert layer moved onto the card adds about half a token per second but costs about 410 MiB the context cache could use.
3. 17 layers in RAM is the fastest setting that loads (33.31 tok/s); 16 doesn't load.
4. The drafter proposes 2 tokens at a time for the 26B to verify.

Results & numbers

• With 21 of 30 expert layers in RAM and the drafter, it holds 131,072 tokens of context, versus 81,920 for the 12B on the same card; without the drafter, 22 layers in RAM hold the full 262,144.
• Generation is 33.7 tok/s with nothing held and 22.8 tok/s with 65,536 tokens held, compared with 55.1 and 30.8 for the 12B with its drafter.
• GSM8K 0.965 and BFCL simple 0.945, level with the 12B's 0.961 and 0.950.
• With every expert in RAM it still generates at 24 tok/s, while a dense 31B with 20 of 60 layers on the GPU manages only 2.5 tok/s.

Gotchas

• The CPU is in the serving path, so thread count matters: 4 threads gave 28.49 tok/s, 6 gave 33.03 and 8 gave 33.97. Other load on the CPU or memory bus slows the model.
• The drafter only gives 1.24x on the 26B against 2.01x on the 12B, and 3 draft tokens loses on sampled text.
• A full GSM8K run took 3 h 26 min versus 1 h 57 min for the 12B for the same score.
• For conversations up to 81,920 tokens the 12B with its drafter is the better pick; the 26B wins when you need more context.

“For a model larger than the card, the active parameter count decides the speed.”— xbill

Do it yourself

1. Build llama.cpp with CUDA and download Google's Gemma 4 26B A4B QAT Q4_0 GGUF and the matching drafter.
2. Or clone github.com/xbill9/gemma4-dev, enter local-llamacpp-3050oem-26b-q4_0, and run make download draft, then make serve.
3. Start with all layers on the GPU (-ngl 99) and use -ncmoe to push expert layers to RAM until the model loads.
4. Trade expert layers for context: use a 4-bit cache (-ctk q4_0 -ctv q4_0) and raise -c until it stops loading.
5. Set -t to your physical core count and benchmark with llama-bench before settling on a setting.

llama-server -m gemma-4-26B_q4_0-it.gguf -ngl 99 -ncmoe 21 -c 131072 -ctk q4_0 -ctv q4_0 -fa 1 \
  -t 8 -tb 12 --parallel 1 --reasoning off --metrics \
  -md gemma-4-26B-A4B-it-qat-assistant-q4_0-exact.gguf --spec-type draft-mtp -ngld all --spec-draft-n-max 2