How to test prompts and LLM outputs with promptfoo, with a CI step

LLM evaluation with promptfoo means writing your prompts, the models to call and a set of test cases in one YAML file, then running promptfoo eval. Each test case carries assertions, such as "output is valid JSON" or "a grader model agrees the answer follows this rubric". The command exits with a non-zero code when tests fail, so the same suite can block a pull request in CI.

promptfoo (opens in a new tab) is an open-source CLI and library under the MIT licence. In March 2026 promptfoo announced it was joining OpenAI (opens in a new tab), and the README now says it is part of OpenAI. The announcement says the open-source tool will stay maintained and keep supporting other providers and models.

Why prompts need regression tests

A prompt is code that a model interprets. Small edits to the wording, a new model version or a different temperature can change outputs on inputs you never looked at. Manual spot checks catch the cases you thought of today and miss the ones you fixed last month.

A regression suite turns those past cases into checks. Every time you edit a prompt or switch models, the suite runs the same inputs and reports which ones broke. promptfoo's CI documentation lists catching regressions before they reach production as the first reason to run evals in a pipeline.

Installing promptfoo

The npm package needs Node.js 22.22.0 or newer, and the docs recommend Node.js 24 LTS. Install it globally, through Homebrew, or run it without installing:

npm install -g promptfoo
# or
brew install promptfoo
# or, with no install
npx promptfoo@latest init --example getting-started

The project README also lists pip install promptfoo. Set the API key for each provider you plan to call:

export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...

promptfoo init starts an interactive setup. promptfoo eval setup opens a browser-based version of the same flow.

A first test suite: prompts, providers and test cases

A promptfoo config has three main parts. prompts holds the templates, with variables in double curly braces. providers lists the models. tests lists the inputs and what each output must satisfy.

The suite below checks a support-ticket triage prompt. The tickets and team names are illustrative sample data.

# prompts/triage.txt
Classify this support ticket. Reply with JSON only, in the form
{"team": "billing" | "technical" | "account", "priority": "low" | "high"}.

Ticket: {{ticket}}

# promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Ticket triage regression suite

prompts:
  - file://prompts/triage.txt

providers:
  - id: openai:gpt-6-sol
    label: gpt-6-sol
  - id: anthropic:messages:claude-sonnet-5-5
    label: claude-sonnet-5-5

defaultTest:
  assert:
    - type: is-json
      value:
        type: object
        required: [team, priority]

tests:
  - description: Double charge goes to billing
    vars:
      ticket: I was charged twice for my March invoice.
    assert:
      - type: javascript
        value: JSON.parse(output).team === 'billing'

  - description: Outage is high priority
    vars:
      ticket: Your API has returned 500 errors for every request since 9am.
    assert:
      - type: javascript
        value: JSON.parse(output).priority === 'high'

  - description: Password reset goes to account
    vars:
      ticket: I can't log in and the reset email never arrives.
    assert:
      - type: javascript
        value: JSON.parse(output).team === 'account'
      - type: not-icontains
        value: sorry

  - description: Reply contains no prose around the JSON
    vars:
      ticket: How do I export my data as CSV?
    assert:
      - type: llm-rubric
        value: The output is a single JSON object with no explanation before or after it.

defaultTest applies its assertions to every test, so each case also has to pass the JSON schema check. Tests can live in separate YAML, JSONL, CSV or JavaScript files and load with tests: file://tests/*.

Run the suite and open the results:

promptfoo eval
promptfoo view

promptfoo eval runs every prompt against every provider and every test case. promptfoo view starts a local web UI with the results grid.

Which promptfoo assertion types should you use?

The promptfoo assertions reference (opens in a new tab) splits assertions into deterministic checks, which run code against the output, and model-assisted checks, which call a model. Any assertion can be inverted by adding not- to its type.

Each assertion has a weight (default 1). A test case can also set its own threshold: the test passes when the weighted average score reaches it. Without one, every assertion must pass.

The grader for llm-rubric and other model-graded checks can be pinned. Pass --grader <provider> on the command line, set provider on the assertion, or set defaultTest.options.provider for the whole suite. Pinning matters in CI, because otherwise the grader is chosen from whichever API keys the environment has.

Comparing two models side by side

Listing two providers, as the suite above does, is the comparison. promptfoo runs each test against both, and the web viewer puts the two outputs for each test next to each other with their pass or fail result. The same works for prompts: list two prompt files and every test runs against both versions.

To try different models without editing the config, override providers on the command line:

promptfoo eval -r openai:gpt-6-sol openai:gpt-6-luna

A few flags help with a fair comparison:

• --repeat <n> runs each test several times, which shows how often a model fails rather than whether it failed once.
• --no-cache forces fresh calls. promptfoo caches provider responses on disk (~/.promptfoo/cache by default), which saves money but hides run-to-run variation.
• --filter-providers <regex> reruns only one model.
• -o results.json (or .html, .csv, .junit.xml) writes the results to a file.

The OpenAI provider docs suggest using a dated model snapshot when one exists to keep the version constant across runs, and note that a fixed snapshot still does not guarantee identical outputs.

Running LLM evaluation with promptfoo in CI

promptfoo eval returns exit code 100 when at least one test fails, and exit code 1 for other errors. Any non-zero code fails a CI step. Two environment variables tune that behavior: PROMPTFOO_PASS_RATE_THRESHOLD sets a minimum pass rate as a percentage (default 100, meaning no failures allowed), and PROMPTFOO_FAILED_TEST_EXIT_CODE changes the exit code for failed tests.

Here is a GitHub Actions workflow built from the commands in promptfoo's CI/CD guide (opens in a new tab):

# .github/workflows/prompt-tests.yml
name: Prompt regression tests
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'promptfooconfig.yaml'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '24'

      - name: Cache promptfoo
        uses: actions/cache@v4
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}
          restore-keys: |
            ${{ runner.os }}-promptfoo-

      - name: Run eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          PROMPTFOO_CACHE_PATH: ~/.cache/promptfoo
          PROMPTFOO_PASS_RATE_THRESHOLD: '95'
        run: >
          npx promptfoo@latest eval
          -c promptfooconfig.yaml
          --grader openai:gpt-6-sol
          -o results.json -o results.junit.xml

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: promptfoo-results
          path: |
            results.json
            results.junit.xml

The JUnit file plugs into CI test-report viewers. --tag git.sha=... attaches pipeline context to the saved eval. For a before-and-after comment on each pull request, promptfoo publishes promptfoo-action (opens in a new tab), which takes the prompt globs, the config path and a GitHub token.

Leave --share out of CI runs that touch real customer data. That flag uploads results to create a shareable URL.

A pre-merge checklist for a prompt suite

• Every bug report about a bad output becomes a test case before the fix.
• Each test has at least one deterministic assertion, so a grader outage cannot pass it.
• The grader model is pinned with --grader or defaultTest.options.provider.
• Latency checks run in a separate job with --no-cache.
• The pass-rate threshold is written down and changed only on purpose.
• Results are uploaded as an artifact so failures can be inspected without rerunning.

What automated evals cannot judge

Assertions check what you thought to check. A suite of 40 passing tests says nothing about input number 41, so coverage depends on how well your cases reflect real traffic.

Model-graded assertions are themselves model outputs. An llm-rubric grader can be wrong, can drift when its provider updates the model, and costs a call per check. promptfoo's docs note that without a threshold, a model-graded check passes on the grader's pass field alone, so {"pass": true, "score": 0} counts as a pass.

Some properties need people. Tone for your brand, whether advice is safe for a specific user, and whether an answer is true for facts the grader does not know all sit outside what an automated check can confirm. promptfoo's getting-started guide says asserts are optional and that many people get value from reviewing outputs manually in the web viewer.

The docs also state that cost and latency assertions check completed responses. They do not cap spending or stop slow requests, so production limits still belong in your application.