What people really do with local, private AI
Real people running AI on their own machines. Real setups with Ollama, LM Studio, llama.cpp and self-hosted tools, sourced from Reddit, Hacker News and blogs, each with a how-to so you can do it yourself.
Rebel BitesHome RAG assistant for embedded-systems docs on Proxmox
Ollama, llama.cpp, Open WebUI, BGE-M3 and SearXNG in a GPU-passthrough VM. Moving from 6 GB to 16 GB of VRAM changed which models could run.
Auto-tagging documents in Paperless-ngx with Ollama
Since version 3.0, Paperless-ngx suggests titles, senders, document types and tags, and can chat with your documents, using a local Ollama model.
Self-hosted bookmarks that tag themselves with local AI (Karakeep)
Karakeep (formerly Hoarder) runs on a Debian LXC, and gemma3:4b in Ollama tags each bookmark. Getting reliable tags from a small model takes one key setting.
Offline meeting minutes with whisper.cpp + Ollama
Turns a 1-hour meeting into structured minutes in under 4 minutes on an RTX 3060, with zero network requests. Bash and Python pipeline included.
Paperless AI on an 8 GB RTX 3060 Ti shared with Frigate
Gemma 3 4B with embeddinggemma answers over 1,318 documents in about 5 seconds, on a GPU that's also running Frigate and Plex.
Overnight video-to-RAG pipeline for 2,000+ saved clips
A local Qwen3.6 35B with ffmpeg, scenedetect and Whisper indexes saved TikToks and YouTube videos, so you can later find 'that one plumbing tutorial'.
Home Assistant Voice PE with a local LLM instead of Echo
A $60 Voice Preview Edition, an N100 box and an optional RTX 3060 replace every Echo and Nest in the house. It keeps working with the internet unplugged.
A private ChatGPT for a family of four on a Mac mini
One always-on M4 Mac mini serves Open WebUI to every phone and laptop in the house. Kids get a tutor preset, and three AI subscriptions are gone.
Hybrid workflow: local for private and repetitive, cloud for hard
A Linux sysadmin stopped trying to replace cloud LLMs. Ollama now handles private docs, overnight article summaries and VS Code autocomplete.
Read-only personal admin assistant over work email and calendar
A project manager juggling a dozen projects has a local bot that builds a wiki from email, surfaces to-dos, sends reminders and drafts replies.
Fully local Home Assistant voice assistant on a GPU
Wake word to spoken answer in 2-3 seconds, with zero audio leaving the house. Built from faster-whisper, Piper and Ollama, with a cloud fallback.
Offline RAG over 268 Obsidian notes, queryable from a coding agent
nomic-embed-text, ChromaDB and mistral-nemo:12b give cited answers from a vault of internal docs, and an MCP server lets the coding agent search it too.
Replacing Google Nest Minis with local Assist on llama.cpp
After a year of Nest Minis getting dumber, a household moved to fully local Assist. Higher-quant GGUF models from Hugging Face are what finally fixed tool calling.
Self-hosted AI email assistant on Linux (n8n + Ollama)
Classifies IMAP mail with llama3.1:8b, routes it, saves reply drafts and extracts action items as JSON. Reports 85-95% classification accuracy.
Extracting clean data from old SEC filings with small models
7B-13B models running locally turn wordy 1990s SEC text filings into structured tables, about 97% accurate on a random sample. Lots of tiny prompts, because tokens are free.
mdrag: a Rust CLI to chat with an Obsidian vault
A self-contained RAG tool built on SQLite with sqlite-vec and gemma3:1b. It answers 'how did I fix that Git submodule issue?' from your own notes.
Connecting a private Obsidian vault to Ollama on macOS
Meeting notes stay off public AI services. llama3.1 and nomic-embed-text in Ollama, wired up through the Local GPT and AI Providers plugins.
Mac Studio as an always-on, LLM-driven smart-home brain
Phrases like 'it's stuffy in here' become real device calls, run by Ollama with qwen3:14b on a Mac Studio. €15 ESP32 satellites keep all audio on the LAN.
Receipt extraction and bank matching with PaperCortex
An open-source local AI layer for Paperless-ngx. It answers plain-English questions, extracts receipt line items, matches bank transactions and exports to DATEV.
Obsidian vault Q&A with Mini-RAG and llama.cpp
An Obsidian plugin that chats with a note or folder, using separate local llama.cpp servers for chat and embeddings (Ollama works too).
Security camera descriptions and search with Frigate GenAI
Frigate sends camera frames to a local vision model (qwen3-vl via Ollama or llama.cpp) for searchable event descriptions, review summaries and chat.
Email triage and morning digest with a local Ollama model (n8n)
Classifies each email as URGENT, REPLY_TODAY, FYI or JUNK using gemma4:e4b, sends urgent alerts and builds a grouped morning digest.
Auto-label Gmail and draft replies with a local model (n8n)
Sorts unread Gmail into six categories with Ollama, applies labels, drafts replies for urgent mail and logs everything to Sheets, with zero paid API costs.
Local-first meeting notes app (whisper.cpp + Ollama)
A macOS app that records your mic and the call's audio as two tracks, transcribes locally and writes a Markdown note with 'You vs. Them' speaker labels.
14 months local-first: what still breaks
An honest report from a three-machine home lab. Local handles about 85% of tasks, but four kinds of work still fail, and that cut cloud spend from $187 to about $25 a month.
Øxygen: a private desktop chat app for Ollama
Conversations, files and settings live in a file in your home directory. It talks only to Ollama on 127.0.0.1 and adds optional embeddings, vision, voice and image generation.
Branching conversation graphs with OpenYoke
A Tauri and Rust desktop app for Ollama where every exchange is a node you can branch from, and sibling branches never bleed context into each other.
Offline chat on desktop or Android via Termux with Prefrontal
A privacy-first chat UI that works with Ollama on desktop or llama.cpp anywhere, including Android through Termux. No telemetry, no subscriptions.
Leaving Google Photos for Immich with local AI search
CLIP smart search and face recognition run on the author's own NAS. A GTX 1070 turned days of indexing into one night.
Offline LLM on a Raspberry Pi 5 for classification and extraction
Phi-3 mini at Q4_K_M runs at 4-6 tokens per second on an 8 GB Pi 5 with no network at all. Under GDPR, data that never leaves the board isn't transferred anywhere.
Aider + qwen2.5-coder:14b as a local Claude Code alternative
A tic-tac-toe test on an RTX 5060 Ti 16 GB. The game worked on the first pass, but the tests and input validation needed several rounds of prompting.
Copilot-style coding assistant from a headless RTX 3090 box
qwen2.5-coder:7b on a home server, reached from a laptop over an SSH tunnel and wired into VS Code through Continue for chat, edits and autocomplete.