Readable Minds: Emergent Theory-of-Mind-Like Behavior in LLM Poker Agents
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
Theory of Mind (ToM) -- the ability to model others' mental states -- is fundamental to human social cognition. Whether large language models (LLMs) can develop ToM has been tested exclusively through static vignettes, leaving open whether ToM-like reasoning can emerge through dynamic interaction. Here we report that autonomous LLM agents playing extended sessions of Texas Hold'em poker progressively develop sophisticated opponent models, but only when equipped with persistent memory. In a 2x2 factorial design crossing memory (present/absent) with domain knowledge (present/absent), each with five replications (N = 20 experiments, ~6,000 agent-hand observations), we find that memory is both necessary and sufficient for ToM-like behavior emergence (Cliff's delta = 1.0, p = 0.008). Agents with memory reach ToM Level 3-5 (predictive to recursive modeling), while agents without memory remain at Level 0 across all replications. Strategic deception grounded in opponent models occurs exclusively in memory-equipped conditions (Fisher's exact p < 0.001). Domain expertise does not gate ToM-like behavior emergence but enhances its application: agents without poker knowledge develop equivalent ToM levels but less precise deception (p = 0.004). Agents with ToM deviate from game-theoretically optimal play (67% vs. 79% TAG adherence, delta = -1.0, p = 0.008) to exploit specific opponents, mirroring expert human play. All mental models are expressed in natural language and directly readable, providing a transparent window into AI social cognition. Cross-model validation with GPT-4o yields weighted Cohen's kappa = 0.81 (almost perfect agreement). These findings demonstrate that functional ToM-like behavior can emerge from interaction dynamics alone, without explicit training or prompting, with implications for understanding artificial social intelligence and biological social cognition.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 43 references that are in the map. First posted on arXiv 5 Apr 2026.
Builds on
22 of 43 references.
References · 22
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
- 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
- 2022Using cognitive psychology to understand GPT-3cited by 389
- 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
- 2018Machine Theory of Mindcited by 222
10 more
- 2023Evaluating Large Language Models in Theory of Mind Taskscited by 217
- 2023Playing repeated games with Large Language Modelscited by 213
- 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 206
- 2020Open Problems in Cooperative AIcited by 167
- 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
- 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
- 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
- 2023Deception Abilities Emerged in Large Language Modelscited by 99
- 2023Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysiscited by 91
- 2023Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framingcited by 49
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics
- 2025Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Task
- 2025The Decrypto Benchmark for Multi-Agent Reasoning and Theory of Mind
- 2025Observer, Not Player: Simulating Theory of Mind in LLMs through Game Observation
- 2025Doing Things with Words: Rethinking Theory of Mind Simulation in Large Language Models
- 2025Understanding Human Limits in Pattern Recognition: A Computational Model of Sequential Reasoning in Rock, Paper, Scissors
- 2026Assessing mentalization in humans and large language models
- 2025Emergent World Beliefs: Exploring Transformers in Stochastic Games