A 4B local model on a 4 GB laptop GPU as a PASS/FAIL test evaluator

What they built

The author's Spring Boot app reads a software incident and suggests its root cause. Because a correct answer can be worded completely differently from the expected one, an exact-match test fails it, so a second language model acts as the evaluator: it gets the expected and the actual root cause and returns PASS or FAIL with a short reason.

The evaluator runs on the laptop rather than on Claude or GPT for two reasons given in the post: every answer the app writes stays on the machine, and a local model is a fixed file pinned by a SHA-256 hash, so a changed score means the app changed, not the evaluator.

Ollama runs in Docker Desktop on Windows 11 with WSL2, Spring AI calls it over HTTP, and qwen3:4b-instruct-2507 (Q4_K_M) does the grading. The write-up measures how much of the model actually fits on a 4 GB card and how long each verdict takes.

Setup

• Dell Precision 7540, Core i7-9850H, 32 GB RAM
• NVIDIA Quadro T1000, 4 GB
• Windows 11 with WSL2 and Docker Desktop; Ollama 0.35.0 in a container with the GPU passed through
• Qwen3 4B Instruct (qwen3:4b-instruct-2507, Q4_K_M), 8-bit KV cache with flash attention
• Java 25, Spring Boot 4.1.1, Spring AI 2.0.1; Postman and Open WebUI for testing

How it works

1. Spring AI's Ollama starter is configured in one YAML file: base-url, pull-model-strategy when_missing, a ten-minute timeout for the first download, temperature 0.1 and num-ctx 4096.
2. The request carries a JSON schema in Ollama's format field with reason before verdict, so the model argues first and decides second, and the Java code reads a field instead of parsing prose.
3. The thinking model qwen3:4b was rejected: it thought for two minutes before answering a one-line question in Open WebUI. The instruct variant answers directly.

Results & numbers

• The model needs 2.94 GB while running; only 2.36 GB (80.4%) landed on the GPU, with 29 of 37 layers on the GPU and 8 on the CPU
• One verdict took 9.82 s: 109 prompt tokens read at 60.4 tokens/s, 125 tokens written at 15.9 tokens/s (about 63 ms per token)
• At that rate one laptop evaluator tops out at about 8,800 verdicts a day; a ~20-token reason would bring it to about 3.2 s per verdict
• The test verdict was correct

Gotchas

• Windows held about 0.8 GiB of video memory and Ollama kept a 457 MiB reserve, so the 4 GB card never offered 4 GB
• Halving the context frees only about 160 MB, not enough to pull the CPU layers back onto the GPU
• Open WebUI added more than 6,000 tokens to a ten-word question, which a 4,096-token context refused
• Ollama unloads a model after five idle minutes by default; the author sends an empty load request before a batch and sets keep_alive to 30 minutes
• If size_vram is smaller than size in /api/ps, part of the model runs on the CPU

“Every answer my app writes stays on my machine.”— Vishnu Vardhan

Do it yourself

1. Run Ollama in Docker with the GPU handed to the container and a volume for downloaded models
2. Pick a non-thinking instruct model from ollama.com's Local filter, e.g. qwen3:4b-instruct-2507
3. Pin temperature (0 or 0.1) and num_ctx (4096) so verdicts and the loaded model stay identical across runs
4. Send the expected and actual answers with a JSON schema in format, reason before verdict
5. Check curl http://localhost:11434/api/ps to see how much of the model is really on the GPU

POST http://localhost:11434/api/chat
{
  "model": "qwen3:4b-instruct-2507-q4_K_M",
  "stream": false,
  "keep_alive": "30m",
  "options": { "temperature": 0, "num_ctx": 4096 },
  "format": {
    "type": "object",
    "properties": {
      "reason":  { "type": "string" },
      "verdict": { "type": "string", "enum": ["PASS", "FAIL"] }
    },
    "required": ["reason", "verdict"]
  },
  "messages": [
    { "role": "system", "content": "You are an evaluator. Compare the candidate root cause with the reference root cause. Answer PASS only if both describe the same underlying cause. Reply in JSON." },
    { "role": "user", "content": "Incident: ...\nReference root cause: ...\nCandidate root cause: ..." }
  ]
}