AI is wrong about your job. Surely it's right about law.

Early on Monday, Gergely Orosz, who writes The Pragmatic Engineer, posted something most heavy AI users will recognise. When he uses AI in an area he knows well, like code on his own project, he spots the model's errors right away and fixes them. When he's outside his field, he trusts it.

His example: he asked three AI models for feedback on a legal case he had also paid a lawyer for. "All 3x models confidently told me X, lawyer then told me Y, and lawyer was right," he wrote. By early afternoon Pacific time the post had passed 93,000 views and 1,700 likes.

He was quoting Vlad Mihalcea, who said Claude Opus 5.5 usually writes code close to how he would, but "every now and then" makes design decisions "that would fail in production."

The old name for this. Michael Crichton called it the Gell-Mann amnesia effect in a 2002 speech, "Why Speculate?" You read a newspaper story about your own field and find it full of errors, then turn the page and read the next story as if it were accurate. Crichton named it after physicist Murray Gell-Mann, and admitted the famous name was there partly to make the idea sound more important.

Our take

The joke in our cartoon is that the same person can be a strict code reviewer and a starry-eyed believer within one browser tab. Swap the newspaper for a chatbot and the effect gets stronger, because the answer is written for you, sounds sure of itself, and arrives in seconds.

Orosz's own rule is a practical one: when you're the expert, use AI to learn and stay sharp. When you're not, assume it may be making things up and you won't notice, so either learn enough to check it or ask a real expert when the stakes are high, like a contract, a tax filing or a medical question.

What we did and didn't do: this story is based on the public posts linked below and Crichton's speech as cited on Wikipedia. We didn't test the legal question ourselves, and Orosz didn't say which models he used.

MODEL Editors