Expert Analysts Followed an AI Agent Random Lean 86 Percent of the Time
Anthropic quietly gave an AI agent a coin flip preference, then watched 50 expert analysts work with it. They matched the agent lean on 86 percent of key decisions. Oversight is weaker than we think.
A coin flip dressed as advice
Here is an experiment every person who reviews AI work should sit with. The Neuron reports that Anthropic secretly gave an AI agent a coin flip preference between two equally defensible options, then watched 50 expert analysts run a farm assessment alongside it. The preference carried no evidence. It was a random lean, plugged into the machine before the work began.
The analysts were not careless beginners. They were experts doing real assessment work, with an agent helping them weigh how to measure productivity, which comparison method to use, which farms the assessment should cover, and how to weight the evidence before a final recommendation. If anyone should be able to hold an independent line against a software suggestion, it is a room like that.
The numbers are not comfortable
Across four key decisions, the analysts matched the agent lean 86 percent of the time. The chart The Neuron published shows the gaps decision by decision. On how to measure productivity, the swing was 82 points: nobody chose option A when the agent leaned toward B, and 82 percent chose it when the agent leaned toward A. The weighting decision swung 78 points, the comparison method 64, and which farms to cover 62.
Only the final recommendation resisted, and only partly, swinging 17 points. In other words, the experts still felt ownership of the headline call, while the quiet choices underneath it, the ones that shape the answer before anyone argues about it, drifted with the machine almost every time. The lean did its work early, where nobody was watching for it.
Why this matters beyond one study
Most of us now work the way those analysts did. A creator asks an assistant which angle leads a story. An editor accepts the cut order a tool suggests. A business owner lets an agent rank leads, draft prices, or sort applications, then reviews the result at the end and calls that oversight. This experiment says the review at the end may be the weakest part of the whole chain.
The risk is not that AI gives loud, obviously wrong answers. Those get caught. The risk is a reasonable sounding lean arriving first, anchoring every judgment that follows, while the human believes the decision stayed theirs. Fifty experts, a random preference, and an 86 percent match rate show how quietly that anchor sets.
How to keep your own judgment in the loop
Three habits fight the anchor. First, write your own call before you ask the tool, even two rough lines, so you have a position to compare instead of a blank page to fill. Second, ask the agent for the strongest case against its own recommendation, and read that case before you read the pitch. Third, spot check one source and one number yourself on anything that ships to a client.
Use AI like the analysts should have: as a fast second worker whose leans you audit, not as the quiet first voice in the room. The tool will keep getting more fluent and more confident. Your edge as a creator or an owner is not typing speed anymore. It is the judgment you can prove was yours.
Sources and further reading
Sources were checked for this article. Tool features, prices and availability can change. Product claims are attributed to their publishers.
