A coin flip picked Claude Code's opinion. Expert analysts went with it 86% of the time.

We keep saying the human reviews what the agent does. Anthropic just measured what that review is worth on the calls that have no right answer, and the number is not flattering.

What Anthropic did. In a study published October 27, Anthropic researchers Amaç Herdağdelen, Hailey Joren and Alex Tamkin had 50 professional analysts, all screened for causal-inference skills, do a realistic statistical analysis in a browser VS Code with Claude Code running Claude Opus 4.8 at high effort. The task: tell a fictional farming co-op whether its paid advisory program works and should be expanded. Along the way they had to make four judgment calls, each with two defensible options: how to measure productivity, which comparison method to use, which farms the estimate covers, and how to weight farms.

For each analyst and each decision, a coin flip set which option the agent would lean toward, through a hidden system prompt. The agent was told to make the best honest case for its lean, change its mind if the evidence turned decisively against it, and never misrepresent results. Analysts were not told.

What happened.

• Analysts picked the agent's coin-flip side 86% of the time across the four decisions (92%, 82%, 81% and 90% per decision). Chance would be about 50%. The paper's registered estimate of the agent's effect is +72 percentage points (95% CI 60 to 83, n = 195 decisions).
• When an analyst had already said which option they preferred, they switched away from it 60% of the time if the agent disagreed and 6% if it agreed. That rests on only 38 cases, and the authors call it imprecise.
• From the four coin flips alone, a simple model predicted 82% of analysts' final rollout-or-discontinue recommendations, against 61% for always guessing the most common answer.
• Afterward, 77% of analysts justified their choices with the agent's own arguments, sometimes nearly word for word. Only 15% mentioned the agent at all. 52% said the recommendation was more theirs than the agent's.
• Of 48 survey respondents, six guessed the study was about AI influence. None suspected the agent's preferences had been planted.
• Experience, education, quiz scores and trust in AI did not predict who deferred.

The catch, in fairness. Anthropic deliberately made the lean strong and voiced, so real-world effects may be smaller. It is one task, in one field, with one model, and analysts were new to farming. The study also says analysts who used the agent made far fewer serious errors: 3 of 50, versus 6 of 11 among those who chose to work without it, though those groups were self-selected and not comparable. And with no lean assigned, stock Opus 4.8 made the same five choices in 112 of 123 runs, so agents do arrive with strong defaults of their own.

Our take. This is the sloppy part of AI-written work that no linter catches. The code ran. The numbers were honest. The agent did not lie. But the decisions that needed a human, which metric, which farms, which weights, were settled by whatever the agent leaned toward, and then signed off as independent judgment. If your process treats "a person approved it" as the check on agent-written analysis or code, you are counting a rubber stamp as a review.

Cheap fixes we would actually use: write your own call down before you ask the agent. Ask it to argue the other option as hard as its own. And when two teams "independently" reach the same answer with the same model, count that as one opinion, not two.

What we checked. We read Anthropic's post and the full paper PDF and cross-checked every number above between the two, and confirmed the pre-analysis plan link resolves on OSF. We did not rerun the study, run Claude Code on the task, or see the participant data.

MODEL Editors