Siesta: a local overnight agent that reads fridge photos and receipts, then sleeps

What they built

Siesta watches one folder on your own laptop. Drop in a photo of your fridge and you get an inventory, what to eat first and a shopping list; a receipt photo becomes a line-by-line row in a local expense ledger; a chat screenshot or export becomes a two-line 'what did I miss' summary; a folder of loose downloads gets a proposed sort, applied only if you ask.

Its rule is that the agent must be cheaper to leave running than to switch off. It skips files whose bytes haven't changed, decides most files from the filename alone at zero tokens, only escalates to the reasoning model when cheaper stages aren't confident, writes one digest per batch, and suppresses a digest that's too similar to the last one.

Everything runs on open-weight models through Ollama on the author's laptop, with no API key, account or upload step.

Setup

• Vision: qwen2.5vl:3b (about 3.0 GB); reasoning: qwen2.5:3b-instruct (about 1.8 GB); embeddings: nomic-embed-text (about 0.26 GB)
• Ollama as a portable install in the user profile, with no service or admin rights
• A laptop with AMD integrated graphics and 12 CPU cores, no discrete GPU
• Python 3.10+ core with no third-party runtime dependencies; FastAPI and uvicorn for the local web UI; one SQLite file in ~/.siesta

How it works

1. A 'nap cascade' routes each file: hash check, then a filename and folder gate, then an optional tiny vision gate, and only then the reasoning model.
2. Every model call passes Ollama a JSON Schema so output is constrained, with a lenient parser only as a safety net.
3. Before writing a digest it embeds the draft and compares it with the last one; above 0.965 cosine similarity nothing new is written.
4. siesta watch runs on a timer and can wait until you've been idle for an hour (--idle-seconds 3600) or stay quiet on battery (--energy-aware).
5. A cost ledger in SQLite records every model call with latency and token count.

Results & numbers

• Over two nights the cascade used 5 model calls and 5,182 tokens versus 12 calls and 8,611 tokens for a naive pipeline, and produced one page instead of six.
• On night one the cascade actually used more tokens (5,182 vs 4,310); the saving is that an unchanged night costs zero tokens.
• On a hand-annotated fridge photo the 3B vision model found all 8 items with nothing invented, in about 21 seconds on CPU.
• On a receipt it read the printed total of $32.24 correctly and warned that the six parsed line items only added up to $23.88.

Gotchas

• Ollama's /api/chat silently ignores a request-level images field, so the model answered blind. The first version's '8 out of 8' fridge score came from the author's own prompt listing the items.
• The giveaway was the token count: 198 prompt tokens instead of about 1,600 for an attached image. Siesta now puts images inside the message and checks on every vision call that the image arrived.
• Never put possible answers in the prompt as examples; a small model will copy them back.
• The 3B vision model isn't good enough to read a crumpled receipt reliably; qwen2.5vl:7b is one flag away.

“In a system that reads the world, you have to prove that it looked.”— Md Faijal Eaqbal

Do it yourself

1. Clone the Siesta repository from GitHub (faijaleaqbal/siesta).
2. Run bash setup.sh on Linux or macOS (or setup.ps1 on Windows) to install a portable Ollama, pull the three models and create a virtualenv.
3. Run python -m siesta init to create the inbox folders and config.
4. Run python -m siesta doctor to confirm the models load and vision actually sees an image.
5. Drop a fridge photo or receipt into the inbox and run python -m siesta scan, or python -m siesta watch for the overnight loop.
6. Open the local UI with python -m siesta serve at http://127.0.0.1:8765.

bash setup.sh
python -m siesta init
python -m siesta doctor
python -m siesta scan
python -m siesta serve   # http://127.0.0.1:8765