Offline RAG over 268 Obsidian notes, queryable from a coding agent

What they built

Alex Beattie's vault holds 268 markdown files: internal docs, architecture decisions, API specs and meeting notes. Built-in search only finds exact terms, and he wanted natural-language questions with synthesized, cited answers, completely offline.

He built a local RAG pipeline and wrapped it as MCP tools, so his coding agent checks the vault instead of him switching context.

Setup

• Embeddings: nomic-embed-text via Ollama (768-dim)
• Vector DB: ChromaDB, persistent on disk
• LLM: mistral-nemo:12b via Ollama
• MCP server: FastMCP (stdio), configured in .cursor/mcp.json
• Code: github.com/alexbeattie/obsidian-rag

How it works

1. Chunk on ## / ### headings, dropping chunks under 150 chars and splitting over 2,000 at paragraph boundaries
2. Embed with task prefixes; IDs are content hashes so re-ingestion upserts
3. Retrieve and synthesize with mistral-nemo, with citations
4. Expose search and Q&A as MCP tools

Results & numbers

• Skipping the nomic prefixes silently drops recall by 15-20%

Gotchas

• nomic-embed-text needs 'search_document: ' and 'search_query: ' prefixes; it cost him an hour of debugging

“No API keys, no cloud services, no data leaving my machine.”— Alex Beattie

Do it yourself

1. ollama pull nomic-embed-text && ollama pull mistral-nemo:12b
2. Split notes on headings with min/max chunk guardrails
3. Prefix documents with 'search_document: ' and queries with 'search_query: '
4. Store in ChromaDB with content-hash IDs; re-run ingest safely
5. Optionally wrap search as an MCP server for your editor's agent