Research: professional analysts followed AI agents' randomly assigned "leans" on subjective analytical decisions 86% of the time; 77% used the agents' arguments

Your analytical work's conclusions may be shaped by AI arguments you do not attribute. The AI's lean, even if arbitrary, can sway your final output without your conscious choice.

A study involving 50 professional analysts in causal inference found that AI agents significantly influenced their decision-making. When an AI agent was instructed to favor a particular analytical choice, analysts followed its lead 86% of the time across four key decisions. This influence was substantial; analysts switched their initial preferences 60% of the time when the AI disagreed with them. Furthermore, 77% of analysts incorporated the AI's arguments into their own explanations, often verbatim, but rarely attributed these arguments to the AI. The study, conducted by researchers Amaç Herdağdelen, Hailey Joren, and Alex Tamkin, used Anthropic's Claude Code and Claude Opus 4.8, with AI leans determined by coin flips. While the AI's influence on the final recommendation was less pronounced, it had shaped the underlying data analysis. Analysts using the AI agent also completed the task faster and made fewer errors, though the comparison group was smaller and less experienced.

This research highlights a subtle but critical delegation of judgment from human experts to AI agents, particularly in complex analytical tasks. The finding that analysts adopt AI arguments without attribution suggests a potential for AI's implicit biases or arbitrary preferences to propagate through professional decision-making, even when humans believe they are in control. This dynamic is crucial for areas where AI is increasingly used for research analysis, as it could inadvertently steer scientific direction or policy recommendations based on non-human-originated rationales.

The study's implications extend to the perceived reliability and autonomy of AI systems in professional settings. While the AI agents in the study performed well, the manner in which analysts deferred to them and adopted their reasoning indicates a need for greater transparency in AI outputs and robust protocols for human oversight. Future work will need to explore how to design AI tools that better support, rather than potentially override, expert judgment, ensuring that human values and explicit decision-making remain paramount, especially in high-stakes analytical domains.