Hardening a two-GPU Ollama server with systemd: 9.2 UNSAFE to 1.4

What they built

Ollama's installer leaves a systemd unit with a user, a restart policy and nothing else: no memory or CPU limit and every device in /dev reachable. systemd-analyze security scored it 9.2 UNSAFE. The author, in the fourth post of an AI homelab series, set out to change that without breaking GPU inference.

Because the install script rewrites ollama.service on every run, all changes live in drop-ins under ollama.service.d/, managed by an Ansible role. Limits were set from measurements of what Ollama's cgroup actually counts, and hardening was applied in six rounds with a regression test and a real model run after each.

The finished unit scored 1.4 OK. The post also hardens a small demo service of the author's own to 0.9 SAFE and deliberately breaks things (kills, OOMs, restart storms) to prove the settings do what they claim.

Setup

• Ubuntu 24.04, kernel 7.0, cgroup v2, systemd 255
• Ollama 0.32.14 listening on 0.0.0.0:11434 for other clients on the server
• Two RTX 5060 Ti 16 GB, 64 GB RAM, 8 GiB swap file
• Models tested include llama3.1:8b and qwen3.8:27b

How it works

1. Limits from measurements: MemoryHigh=16G, MemoryMax=32G, CPUQuota=1600%, TasksMax=512, Restart=on-failure, OOMPolicy=continue and a start limit of 5 starts in 5 minutes.
2. ProtectSystem=strict with ~/.ollama and ~/.nv left writable; DevicePolicy=closed allowing only the char-nvidia and char-nvidia-uvm groups, with nvidia_uvm loaded first via modprobe@nvidia_uvm.service.
3. Later rounds add an empty capability set, kernel and process protections, RestrictAddressFamilies, SystemCallFilter=@system-service with ~@privileged, PrivateUsers and MemoryDenyWriteExecute, each tried alone in a temporary drop-in first.

Results & numbers

• Exposure score went from 9.2 UNSAFE to 1.4 OK
• Loading two models (~22 GiB of files) stopped the cgroup 3.2 MiB below MemoryHigh; the kernel reclaimed cache and nothing was killed
• The CPU quota never throttled a CPU-only model (13.2 tokens/s with it, 13.6 without)
• Final tests passed: pulls, embeddings, a Docker client, a model on both cards at 100% GPU, vision, a CPU-only model and a reboot

Gotchas

• Most of Ollama's cgroup memory is page cache from model files, even after unloading: 23 GiB counted where Ollama needed about 1.4 GiB
• MemoryMax without MemorySwapMax=0 limits RAM, not memory: the service just grows into swap
• PrivateDevices hides /dev/nvidia*, and a GPU service without its devices silently runs on the CPU; check the log for the cards after every change
• ProcSubset=pid broke Ollama's /proc/meminfo lookup with only one warning in the log
• systemd's default start limit never triggers with Ollama's 3-second restart delay, so a broken Ollama would restart forever

“A service without its GPU devices does not fail; it waits, then runs on the CPU.”— Gancho Stoychev

Do it yourself

1. Run systemd-analyze security ollama.service to get a baseline score
2. Measure memory.current, page cache and anonymous memory with real models loaded before choosing limits
3. Put every change in a drop-in under /etc/systemd/system/ollama.service.d/, never in the installer's unit
4. Add limits, restart and OOM policy, then ProtectSystem=strict and a closed device policy allowing only the NVIDIA devices
5. Add each risky directive alone in a temporary drop-in under /run, run a real model on the GPU, and keep it only if it passes

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
ExecStartPre=+/usr/bin/timeout 60 /bin/sh -c 'until /usr/bin/nvidia-smi -L >/dev/null 2>&1; do sleep 1; done'