Best Tools · Coding
promptfoo 0.125
SummaryThe open-source tool for testing prompts, agents and RAG apps got half as heavy to install and learned to tell you which test is flaky.
Filed October 23, 2026. Price, as published by the maker: Free, open source (MIT). Local evals need no account; model tests need your own provider keys..
Open the maker’s site (opens in a new tab)
promptfoo runs your prompts, agents or RAG pipeline against a list of test cases from a YAML file and checks each answer with assertions: exact match, contains, JavaScript or Python checks, similarity scores, AI graders and more. It also does red teaming. It runs from the command line and in CI. Promptfoo is now part of OpenAI and stays open source under MIT. Version 0.125.0 shipped October 23, 2026.
How we tested. October 23, 2026, on an 8-core Linux box with Node 22. We installed 0.125.0 and the previous 0.124.1 side by side and ran the same small eval files through both, using promptfoo's built-in echo provider and our own local test provider, so no AI model and no API key. We checked the install size, the new Unicode option, the new repeat-stability report, the new --safe-mode flag and the new rouge-l check. The full test is in the story.
Pro: 0.125.0 installed at 618 MB against 1,252 MB for 0.124.1, because the Claude Agent and Codex SDKs are now opt-in. With --repeat 10 it flagged the one question our test provider got wrong 3 times in 10, with a 95% confidence range, where 0.124.1 only gave a total.
Con: --safe-mode blocked inline JavaScript checks, but our JavaScript provider file and an inline Python check still ran, as the docs warn ("not a sandbox"). Unicode matching is off by default, so "café" typed two ways still fails unless you add normalizeUnicode: true. Users of the Claude Agent or Codex providers now have to install those SDKs themselves.