Replacing Google Nest Minis with local Assist on llama.cpp

What they built

Google Assistant on Nest Minis kept getting worse, the household didn't like always-online microphones, and cloud outages broke light control. So the author moved to Home Assistant Assist backed by a local model.

They started with Ollama and moved to llama.cpp, then layered on integrations for weather, web search and place search so Assist could actually replace the speakers.

Setup

• Runner: llama.cpp (previously Ollama)
• Models tested include GPT-OSS 20B MXFP4, Qwen3.5-35B-A3B MXFP4_MOE, Qwen3-VL 8B Q6_K_XL, Qwen3-30B-A3B Q4_K_XL, GLM 4.7 Flash Q4_K_XL, Qwen3 4B Instruct Q6_K_XL
• STT: Nvidia Parakeet V2 via Wyoming ONNX ASR (OpenVINO branch)
• TTS: Kokoro TTS or Piper
• Home Assistant Voice Preview Edition for testing
• HA integrations: LLM Conversation, LLM Intents (web search, place search, weather)

Results & numbers

• Parakeet V2 on the OpenVINO branch brings CPU speech-to-text down to about 0.3 seconds
• Piper handles general text but struggles with currency, phone numbers and addresses

Gotchas

• Ollama's default :4b tags are often low quantization (Q4_K), which broke basic tool calls
• Wi-Fi issues made the Voice PE's speech stutter; streaming Piper and a dedicated IoT network fixed it

“Once I learned about the ability to use HuggingFace to find GGUF models with higher quantizations, assist was immediately performing much better”— NOW LET US author

Do it yourself

1. Run llama.cpp (or Ollama) with a higher-quantization GGUF from Hugging Face rather than a default Q4 tag
2. Connect it to Home Assistant Assist and test basic tool calls first
3. Use Wyoming ONNX ASR with Parakeet V2 for fast CPU speech-to-text
4. Pick Kokoro or Piper for speech output, with streaming enabled
5. Add the LLM Intents integration for weather, web and place search