Chat-based Large Language Models replicate the mechanisms of a psychic's con

The tools you use to generate text may be making you believe they understand you more than they do.

Baldur Bjarnason's research suggests that the perceived intelligence of chat-based Large Language Models (LLMs) is an illusion, mirroring the techniques used by psychic con artists. Bjarnason argues that LLMs, which are essentially mathematical models of language tokens, do not possess genuine intelligence or reasoning capabilities. Instead, they generate statistically plausible responses based on input text. This effect, termed the "LLMentalist Effect," arises from the user's mind, not the LLM's inherent intelligence. The mechanism closely resembles "cold reading," where generic statements are presented as specific insights through validation statements and the Forer effect. This leads users to believe the LLM is understanding them personally, much like a psychic convinces a client they are being uniquely

This phenomenon highlights a critical gap between user perception and LLM functionality, suggesting a need for greater transparency in how these models operate. The illusion of intelligence could lead to over-reliance on LLMs for tasks requiring genuine understanding or critical thinking. As LLMs become more integrated into various industries, this "LLMentalist Effect" could have significant implications for user trust and the responsible deployment of AI technologies. It implies that current LLM architectures may not be on a path to true artificial general intelligence, but rather are sophisticated pattern-matching machines skilled at mimicry.

The research implies that companies developing and deploying LLMs must address this user-centric illusion to avoid potential misuse or disappointment. Understanding this mechanism is crucial for setting realistic expectations and designing interfaces that do not inadvertently foster anthropomorphism. Future developments may focus on ways to mitigate this effect, either by designing LLMs that are less prone to generating such convincing illusions or by educating users on the inherent limitations of current AI.