Fine-tuning Gemma 4 E2B on an 8 GB RTX 3050 so it thinks less

What they built

xbill set out to answer whether a used 8 GB inference card can also fine-tune, or whether training needs rented hardware. The goal was a 'lazy' Gemma 4 E2B that reaches the same answer to grade-school maths problems with less thinking.

The method: ask the model each training problem four times, keep the shortest right answer, and fine-tune on those. Because the deliverable is a 4-bit file for small cards, he wrote a trainer that trains through the exact Q4_0 weights the model is served from, plus an exporter that writes the trained weights back into that same GGUF file.

Everything, from sampling to training to grading, ran on one RTX 3050 through llama.cpp, and every log, script and number is committed in the lazy-gemma repository.

Setup

• NVIDIA GeForce RTX 3050 OEM, 8,192 MiB, in a desktop with 46 GB of RAM
• llama.cpp built with CUDA, serving Gemma 4 E2B with Google's quantization-aware weights as exact 4-bit Q4_0 (about 109 tokens per second)
• Python 3.14 with torch 2.14.1 and transformers 5.12.1; peft and bitsandbytes for a standard QLoRA comparison
• GSM8K: 7,473 training problems and a fixed held-out set of 250 test problems, graded by code

How it works

1. sample_train.py asked the served model 1,000 training problems four times each at temperature 1.0 on a 4-slot llama-server, keeping exact token ids.
2. qat_lora.py reads every 4-bit weight out of the GGUF, adds a rank-16 LoRA adapter (24.2M values) and rounds the sum back onto the same 4-bit grid each forward pass.
3. To fit in 8 GB, weights stay 4-bit in memory, Gemma 4's per-layer embedding table stays on the CPU, and gradient checkpointing and chunked loss keep the peak under 5 GB.
4. export_gguf.py copies the served GGUF and overwrites only the trained 4-bit weights, so the result loads in the same llama-server with the same flags.

Results & numbers

• The full loop took 6 h 44 min: 4 h 1 min to sample 3.7 million tokens, 2 h 10 min to train at a 4,896 MiB peak, and 32 min to grade.
• The untrained model got 227 of 250 right at 977.8 tokens per correct answer.
• The best tuned run (lr 1e-3) cut tokens per correct answer by 6.3% with one fewer right answer, and beat the standard QLoRA recipe on the same data by 4.9% fewer tokens.
• llama.cpp's built-in thinking cap of 256 tokens did better with no training at all: 228 right and 35.8% fewer tokens per correct answer.

Gotchas

• llama.cpp's own trainer stopped on Gemma 4 at the tested commit with 'unsupported unary op for backward pass: TANH', and llama-finetune only trains full weights.
• Hugging Face's Trainer puts the whole model on the GPU, including Gemma 4's 4.7 GB per-layer table, so the first backward pass ran out of memory until that table was moved to the CPU.
• At lr 1e-4 the adapter trained for hours but changed only 0.013% of the 4-bit weights, because corrections smaller than half a 4-bit step round away. Count the weights that actually moved.
• Sampling the training data, which is pure inference, took longer than the training itself.
• Try the free fix first: a thinking cap on each request beat every fine-tune here.

Do it yourself

1. Build llama.cpp with CUDA and serve Gemma 4 E2B in its QAT Q4_0 GGUF on your card.
2. Clone github.com/xbill9/lazy-gemma and run make_splits.py to fix the training and held-out sets before anything trains.
3. Measure the baseline and llama.cpp's thinking_budget_tokens cap with lazy_eval.py; that is the number a fine-tune has to beat.
4. Sample several answers per training problem with sample_train.py, then train on the shortest right answer with train.py.
5. Export with export_gguf.py and re-run lazy_eval.py against the tuned file in the same llama-server.

git clone https://github.com/xbill9/lazy-gemma