Four 125B-748B open models on one RTX 5090 and 128 GB of RAM, benchmarked side by side

What they built

Over several weeks Seppe Gadeyne got four large mixture-of-experts models running on the same workstation, each on its own runtime: Qwen3.8-Flash-Next on Strata, Ornith-1.5-122B on TabbyAPI with ExLlamaV3, and GLM-5.3-Flash and DeepSeek-V4.1-Flash on Project Maya. Then he measured all four in one session with the same benchmarks.

The models range from 125B to 748B parameters. They fit on one 32 GB GPU because only part of each model works on each token, and the runtimes keep the most-used experts in VRAM, the rest in RAM, and in DeepSeek's case read part of the model from the SSD while it generates.

Besides speed and long-context tests, he scored each model on nine tasks from his team's real work, such as WordPress PHP snippets, SEO meta tags, LocalBusiness JSON-LD and a Dutch blog intro, each checked automatically. The result is a practical guide to which local model they hand which job.

Setup

• GPU: RTX 5090 with 32 GB VRAM, also driving the desktop; every runtime leaves 3 GiB free
• CPU: Ryzen 9 9950X3D (16 cores, 32 threads); RAM: 128 GB DDR5 (4x32 GB at 5,600 MT/s); one NVMe drive
• OS: Arch Linux (Omarchy), NVIDIA driver 615.78, CUDA 13.4
• Qwen3.8-Flash-Next (Unsloth UD-Q5_K_XL GGUF) on Strata, 512K context; Ornith-1.5-122B (EXL3 4 bpw) on TabbyAPI + ExLlamaV3, 256K; GLM-5.3-Flash (EXL3 3.05 bpw) on Project Maya, 512K; DeepSeek-V4.1-Flash (Maya-S, experts in 2-3 bits) on Project Maya, 262K

How it works

1. Each model ran alone, started by its own script in its own systemd slice without a memory limit, so shell limits or systemd-oomd could not interfere.
2. Decode speed came from a streaming client against each OpenAI-compatible endpoint: eight runs alternating a Dutch prose prompt and a coding prompt, 512 output tokens, median reported.
3. A needle-in-a-haystack test with roughly 31K and 125K token prompts, with a new random seed each run, measured prefill speed and long-context recall.
4. Quality was nine day-job tasks with automatic checkers, 54 points per round, two rounds, the second in reverse order.

Results & numbers

• Decode speed: Qwen 78.8 tok/s, Ornith 46.4, GLM 30.1, DeepSeek 24.9.
• Prefill on the ~125K prompt: Qwen 4,252 tok/s (29 seconds), DeepSeek 2,570, GLM 2,445, Ornith 1,508 (87 seconds). All four found the needle at both lengths.
• Quality: Qwen and DeepSeek scored 108/108, GLM 98/108, Ornith 83/108. Qwen finished a round of nine tasks in 104 seconds, DeepSeek in 680.
• Qwen on Strata became their default for coding and agent work, with GLM as a second opinion and DeepSeek for hard problems where they can wait.

Gotchas

• Only one model fits at a time: Strata and Maya with GLM pin almost all 128 GB of RAM, and switching takes a stop and a start script.
• DeepSeek on Maya reads its files in place, so free reports about 100 GiB available that is really page cache the engine depends on; starting anything else pushes experts back to the SSD.
• Ornith's misses were often stray tokens inside otherwise correct output, such as a Chinese word inside a JSON object, so they read its answers before running them.
• Drafting is not always a win: with most of Ornith's experts on the CPU, no drafting (45.2 tok/s) beat its MTP head (37.8) and the DFlash2 drafter (40.6).
• The runtimes move fast: Project Maya 1.1.21 made DeepSeek's long-prompt reading about 3.7 times faster than the version measured a day earlier.

“They do tell you which ones we can hand our everyday work to without checking every answer.”— Seppe Gadeyne

Do it yourself

1. Start with a GPU with lots of VRAM and as much fast system RAM as you can, since the experts that don't fit on the card live in RAM.
2. Pick a mixture-of-experts model and a runtime that splits experts between VRAM and RAM; the author links step-by-step setup guides for Qwen on Strata, GLM on Maya and DeepSeek on Maya.
3. Leave some VRAM free if the same GPU drives your desktop (the author reserves 3 GiB).
4. Run each runtime in its own systemd slice so memory limits and systemd-oomd don't kill it.
5. Benchmark with your own tasks and automatic checkers, not just tokens per second, and count tokens from the API's usage numbers rather than stream chunks.