Your AI fixed the bug. Trail of Bits counted what else broke.

Today was the day AI found the bugs: two critical PHP flaws, disclosed through Patch the Planet, which security firm Trail of Bits calls its joint initiative with OpenAI. The less glamorous question is what happens when an agent writes the fix. Trail of Bits answered it with its own numbers.

What they counted. In a post published Wednesday, Trail of Bits engineers Anish Naik and Marcelo Morales say they reviewed every Patch the Planet patch a Trail of Bits engineer authored and maintainers merged upstream. These patches were written by agents, with engineers directing the work and checking it. The review confirmed 21 regressions across more than 130 merged pull requests: 16 functional, four build or test, one performance. "None broke a security protection," they write. A regression here means a new defect that breaks behavior that used to work, on a path a real user or attacker can reach.

Finding them was noisy too. An agent hunted for the regressions and had to confirm each with a proof of concept run before and after the change. In a single run it flagged 58 candidates, and 21 held up, about 36% precision. Engineers sorted the rest by hand.

The fix they shipped. Trail of Bits released two plugins in its open trailofbits/skills repo. regression-hunt traces a diff to the code's real interfaces, reproduces each suspected regression on a running build, replays it with a fresh agent, and only reports it if it passes a strict definition. Against a plain Codex agent told to "review the PR for regressions," it produced 25% fewer false positives on the 21 known regressions (95% confidence interval 14% to 38%), but left some real ones marked borderline. patching-bugs requires a "three-arm proof": the same reproduction must fail on the original code, pass on the patch, and fail again with the patch fully reverted. It never pushes or opens a PR itself.

The human baseline. In September, the same team reported that across 2,265 vulnerabilities from its client audits in 2024 to 2026, developers' first fixes failed to fully resolve the issue 283 times, or 12.5%, roughly one in eight, even with a detailed report in hand. That's a different measure from regressions, so it isn't a like-for-like comparison, and Trail of Bits says as much. In one case it describes, a human maintainer and an agent independently made the same crash-causing mistake in the same freenginx fix.

Our take

The scary number isn't 21. It's how few teams could tell you their own. Agent-written patches are already landing in projects you depend on, and the people best placed to measure them found 21 regressions in 130-plus merges, and their regression-hunting agent was wrong almost two times in three. That's not a case against AI patches; humans miss too. It's a case against accepting any patch, from anyone, on vibes. Make the fix prove itself: a repro that fails before, passes after, and fails again when you revert. Write down what counts as a regression in your project. If an agent can't produce the evidence, the patch isn't done.

What we did and didn't do: we read both Trail of Bits posts in full and checked that both plugins and their READMEs are public on GitHub (added October 20). The counts are Trail of Bits' own, about its own work. We didn't run the plugins or check any of the regressions ourselves. Not sponsored, no affiliate links.

MODEL Editors