Fixing Ollama's defaults for small models in a desktop chat app

What they built

Micah Spiegel, who comes from finance rather than CS, is building Qwellar, a desktop app that treats a local AI as a long-term presence: persistent memory, chat with local models via Ollama, and a workspace for working with AI over time. It talks to llama3.2:3b through Ollama's API on localhost:11434.

The app teaches the model code words in the system prompt, such as outputting a confirm_start_fresh tag when the user asks to wipe the chat; the app watches for the tag and shows a confirmation card. At Ollama's default temperature of 0.8 the model wrote variations of the tag or paraphrased it, and the parser failed.

The default 2048-token context also silently dropped the start of longer conversations, so the model made things up about earlier messages. The post walks through the settings changes, a bug that kept the options from ever reaching Ollama, and the code involved.

Setup

• Ollama on localhost:11434 with llama3.2:3b (about 2 GB of RAM)
• Qwellar: Electron desktop app with a TypeScript Ollama client

How it works

1. A BASELINE_MODEL_OPTIONS constant (temperature 0.15, num_ctx 16384) is merged into every request in the Ollama client, with caller options able to override it.
2. The Electron main process was forwarding only model and messages; a one-line fix passes options through too.
3. Prompts were hardened with explicit output-only instructions and labelled negative examples of bad output the model had produced.
4. The parser now tolerates small mistakes like extra spaces, trailing text or markdown wrapping.
5. A Settings UI exposes response style (Precise 0.15, Balanced 0.4, Creative 0.8, Custom) and context window (4k to 32k or model default).

Results & numbers

• At temperature 0.15 small models stopped mangling structured output and the confirmation tag fired correctly
• num_ctx 16384 gives about 80 turns of chat before running out, versus about 10 at 2048

Gotchas

• Ollama does not report when a conversation exceeds num_ctx; it silently cuts off the beginning
• The options bug returned HTTP 200 every time, so a status-code check never caught it; only end-to-end testing did
• Hardened prompts mean paying maintenance cost in English instead of code for every new feature

“Silence equals defaults. Most people never even know the field exists.”— Micah Spiegel

Do it yourself

1. Add an options object to every /api/chat request instead of relying on defaults
2. Use a low temperature (e.g. 0.15) for anything your code parses
3. Raise num_ctx (e.g. 16384) to fit whole conversations within your RAM budget
4. Verify the options actually reach Ollama end to end, not just that the API returns 200

POST http://localhost:11434/api/chat
{
  "model": "llama3.2:3b",
  "messages": [
    { "role": "user", "content": "hey what's up" }
  ],
  "options": {
    "temperature": 0.15,
    "num_ctx": 16384
  }
}