PromptCI: regression-testing LLM prompts offline against Ollama

What they built

PromptCI is a command-line CI tool for LLM workflows. The author's example: a freelancer edits a prompt on Friday evening, and by Monday the client's bot is writing 'TODO' in customer emails. PromptCI is meant to catch that kind of silent drift before users do.

You describe a workflow in YAML, keep prompts in separate Jinja2 template files, and attach checks to the output. After changing a prompt you run one command and it tells you whether behaviour changed.

By default it talks to Ollama, so every test run uses open-weight models on your own machine. The author's reasoning is that testing prompts on every change only makes sense if each run is free, and prompts often contain real customer text that shouldn't leave the machine.

Setup

• Python package installed from GitHub (version 0.1.0, MIT-licensed)
• Ollama running open-weight models such as Llama, Qwen, Mistral or Gemma
• YAML pipelines with Jinja2 prompt templates

How it works

1. Checks include contains, regex, length, JSON schema, and an optional LLM judge that scores the answer.
2. Every output is saved as a snapshot, and --frozen mode fails the run if anything has changed.
3. Clean exit codes let it plug into any CI system, including GitHub Actions.
4. Swapping the model is a one-line change in the YAML.

Results & numbers

• The whole workflow runs offline with no API keys, and the project's own test suite runs offline using a built-in mock model

Gotchas

• It's an early 0.1.0 release; the author says more features are coming

“Now go test your prompts before your users do.”— Soumyabrata Mukherjee

Do it yourself

1. Install Ollama and pull an open-weight model you use in production prompts.
2. Install PromptCI with pip from its GitHub repository.
3. Run promptci init to scaffold a workflow, then describe your pipeline and checks in YAML.
4. Run promptci run to record outputs, then promptci run --frozen after each prompt edit to catch drift.
5. Add the frozen run to your CI so a changed answer fails the build.

pip install git+https://github.com/Soumyabrataop/promptci.git
promptci init
promptci run
promptci run --frozen