promptfoo 0.125, tested: half the install, and it finally names the flaky test

promptfoo is the open-source tool a lot of AI teams use to test their prompts, agents and RAG apps before shipping: you write test cases in a YAML file, it runs them against one or more models, and it checks every answer with assertions. It also does red teaming. It has more than 26,000 GitHub stars, agreed in March to be acquired by OpenAI (its README now says it is part of OpenAI), and stays open source under MIT. It was on GitHub's TypeScript trending list this morning.

Version 0.125.0 came out today at 8:32 a.m. Pacific. The release notes list two breaking changes (the Claude Agent and Codex SDKs are now opt-in, and jks-js is an optional dependency) and new features including Unicode normalization for text matching, a --safe-mode flag, a report on unstable results across repeated runs, and ROUGE-L and ROUGE-S scoring.

The test. We installed 0.125.0 and the previous release, 0.124.1, side by side from npm with Node 22 on an 8-core Linux box (the package now needs Node 22.22 or newer). npm verified registry signatures and provenance for the package. Telemetry and update checks were off. Every eval used promptfoo's built-in echo provider or a small local test provider we wrote, so no AI model, no API key and no network calls. We ran the same files through both versions.

Install size. 0.125.0 took 618 MB on disk against 1,252 MB for 0.124.1, about half. The old version ships the Codex command-line binary (370 MB) and the Claude Agent binary (231 MB) to everyone. The new one doesn't, unless you install those SDKs yourself. Start-up time didn't change: about 1.2 to 1.5 seconds for promptfoo --version on both.

Unicode. We fed it "café crème" with the accent stored as a separate character, and asked it to match "café crème" with the accent built in. They look identical and have different bytes. By default both versions fail it, with a message that reads, literally, Expected output "café crème" to equal "café crème". With the new normalizeUnicode: true, 0.125.0 passed both the equals and the contains check. 0.124.1 ignored the option with no warning and failed. The stricter mode is opt-in on purpose: plain true still treats x² and x2 as different, and NFKC treats them as the same.

Flaky answers. Models don't give the same answer every time, so one pass can lie. We wrote a test provider that answers "What is the capital of France?" wrong every third time, plus one question it always gets right and one it always gets wrong, and ran each 10 times with --repeat 10. Both versions reported the same total: 17 passed, 13 failed. Only 0.125.0 said why: "1 unstable: 7/10 passed across 10 runs; 95% CI 39.7%–89.2%." The JSON output lists every group with its pass, fail and error counts. One catch: without a test description it names the flaky group by the prompt template ({{q}}), which doesn't tell you which question it was. Give each test a description.

Safe mode. Promptfoo configs can run code, which matters when you run a config someone else wrote. With --safe-mode, 0.125.0 refused an inline JavaScript check ("Inline JavaScript execution is disabled in safe mode") and still ran the same check from a file. Setting PROMPTFOO_SAFE_MODE=true held even when the config file tried to switch it off. But our JavaScript provider file ran, and an inline Python check ran and passed. That matches the docs, which call it a "limited guard, not a sandbox." It is a seatbelt for one kind of inline code, not a way to run untrusted configs.

ROUGE-L. The new rouge-l check scores word order within sentences. Against the reference "DuckDB 2.0 is faster. It cannot open old files.", a close paraphrase scored 0.615 and passed our 0.5 bar. The same words scrambled backwards scored 0.348 and failed, while the older rouge-n check gave the scramble 0.762 and passed it. 0.124.1 didn't know the type.

The catch. Safe mode is narrower than its name. Unicode matching is off unless you ask. And if you use promptfoo's Claude Agent or Codex providers, the upgrade now means installing those SDKs yourself.

What we did not test. Real model providers, AI graders and red teaming (they need API keys), the new cache-hit count in the summary, ROUGE-S, trace-span filters, provider-only environment settings, the opt-in SDK install path, the web viewer, Docker, Windows and macOS.

The verdict. Worth the upgrade for anyone running evals in CI: half the install, and a repeat report that tells you which test is flaky instead of hiding it in a total. Turn on normalizeUnicode for text from real users, and don't treat --safe-mode as a sandbox.

Disclosure: not sponsored. No affiliate links. MODEL has no relationship with promptfoo or OpenAI. Full test notes, commands and outputs are kept by MODEL Editors.