A coin flip picked the AI agent's opinion. Expert analysts went with it 86% of the time

The usual pitch for AI agents at work: the agent writes the code and runs the numbers, and the expert makes the calls that matter. Anthropic tested whether that's actually what happens. Mostly, it isn't.

The setup. In a study published today (opens in a new tab) by Amaç Herdağdelen, Hailey Joren and Alex Tamkin, 50 professional analysts, all experienced in causal inference, used Claude Code (running Claude Opus 4.8 at High effort) to advise a fictional farming cooperative on whether to expand a paid advisory program. Along the way they had to make four choices, each with two reasonable options: measure productivity as yield per hectare or yield per unit of operating cost; which comparison method to use; which farms the estimate covers; and whether to weight bigger farms more. Then they made a final recommendation.

For every analyst and every decision, a coin flip set which option the agent would lean toward. The analysts weren't told. The agent was instructed to make the best honest case for its lean, change its mind if the evidence demanded it, and never mislead.

What happened.

• Across the four analytical choices, analysts went with their agent's lean 86% of the time. With no influence you'd expect about 50%. Per decision: 92% (outcome measure), 82% (comparison method), 81% (which farms), 90% (weighting).
• When an analyst had already stated a preference and the agent then disagreed, they switched 60% of the time, versus 6% when the agent agreed.
• The effect on the final recommendation was weaker: 54% recommended rolling the program out when the agent leaned that way, versus 36% when it leaned the other way. But using only the coin flips on the four analytical choices, a statistical model predicted 82% of recommendations (61% for always guessing the most common answer).
• Afterward, 77% of analysts repeated the agent's arguments when explaining their choices, sometimes nearly word for word. Only 15% mentioned the agent at all. None suspected the agent's preferences had been set on purpose.

The fine print, from Anthropic itself. The leans were deliberately strengthened so they could be measured; with no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, and real-world effects may be smaller. It was one task, in one field, with one model, and the analysts were new to farming. The study recorded which option people picked, not how strongly they felt. And it's Anthropic studying its own product; it hasn't been independently replicated.

It's also not an anti-AI result. Of the 11 participants who chose to work without the agent, six made serious methodological errors, versus three of the 50 who used it, and they took longer (median 90 minutes versus 65). Anthropic notes that group was smaller and less experienced, so the comparison is shaky.

Our take. The agent didn't lie to anyone. It just argued one side well, and experts adopted that side as their own. If you use agents for analysis, ask for the strongest case for each option before you pick, or write down your own call before you ask. A sign-off only counts as a second opinion if it's actually yours.

Sources: Anthropic's research post and paper, read October 27, 2026. MODEL did not run this experiment. Not sponsored; no affiliate links.

MODEL Editors