# Large Language Models Assume People are More Rational than We Really are

arXiv: https://arxiv.org/abs/2406.17055 · HTML: https://mlsanity.com/paper/2406.17055/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 80 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 30 of 45)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [Quantifying the Persona Effect in LLM Simulations](https://mlsanity.com/paper/2402.10811/) (2024)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Teaching Large Language Models to Reason with Reinforcement Learning](https://mlsanity.com/paper/2403.04642/) (2024)
- [In-Context Impersonation Reveals Large Language Models' Strengths and Biases](https://mlsanity.com/paper/2305.14930/) (2023)
- [Human heuristics for AI-generated language are flawed](https://mlsanity.com/paper/2206.07271/) (2022)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [Turning large language models into cognitive models](https://mlsanity.com/paper/2306.03917/) (2023)
- [CogBench: a large language model walks into a psychology lab](https://mlsanity.com/paper/2402.18225/) (2024)
- [Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs](https://mlsanity.com/paper/2403.05020/) (2024)
- [Rehearsal: Simulating Conflict to Teach Conflict Resolution](https://mlsanity.com/paper/2309.12309/) (2023)
- [Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information](https://mlsanity.com/paper/2402.18144/) (2024)

## Built on by (most cited first, 30 of 54)

- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse](https://mlsanity.com/paper/2410.21333/) (2024)
- [SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs](https://mlsanity.com/paper/2410.13648/) (2024)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models](https://mlsanity.com/paper/2503.17523/) (2025)
- [Accumulating Context Changes the Beliefs of Language Models](https://mlsanity.com/paper/2511.01805/) (2025)
- [On Benchmarking Human-Like Intelligence in Machines](https://mlsanity.com/paper/2502.20502/) (2025)
- [EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety](https://mlsanity.com/paper/2504.09689/) (2025)
- [When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings](https://mlsanity.com/paper/2407.04503/) (2024)
- [FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights](https://mlsanity.com/paper/2602.02905/) (2026)
- [Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems](https://mlsanity.com/paper/2505.17968/) (2025)
- [From Perceptions to Decisions: Wildfire Evacuation Decision Prediction with Behavioral Theory-informed LLMs](https://mlsanity.com/paper/2502.17701/) (2025)
- [A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments](https://mlsanity.com/paper/2509.25609/) (2025)
- [Levels of Analysis for Large Language Models](https://mlsanity.com/paper/2503.13401/) (2025)
- [MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation](https://mlsanity.com/paper/2508.14146/) (2025)
- [When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs](https://mlsanity.com/paper/2602.06286/) (2026)
- [The Double-Edged Sword of Behavioral Responses in Strategic Classification: Theory and User Studies](https://mlsanity.com/paper/2410.18066/) (2024)
- [Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values](https://mlsanity.com/paper/2502.00313/) (2025)
- [Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models](https://mlsanity.com/paper/2604.02485/) (2026)
- [Are Large Language Models Sensitive to the Motives Behind Communication?](https://mlsanity.com/paper/2510.19687/) (2025)
- [HARBOR: Exploring Persona Dynamics in Multi-Agent Competition](https://mlsanity.com/paper/2502.12149/) (2025)
- [Steering Risk Preferences in Large Language Models by Aligning Behavioral and Neural Representations](https://mlsanity.com/paper/2505.11615/) (2025)
- [Cognitive LLMs: Towards Integrating Cognitive Architectures and Large Language Models for Manufacturing Decision-making](https://mlsanity.com/paper/2408.09176/) (2024)
- [LLM Agents Are Hypersensitive to Nudges](https://mlsanity.com/paper/2505.11584/) (2025)
- [Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities](https://mlsanity.com/paper/2603.10396/) (2026)
- [Can Revealed Preferences Clarify LLM Alignment and Steering?](https://mlsanity.com/paper/2605.08556/) (2026)
- [Evaluating Language Models' Evaluations of Games](https://mlsanity.com/paper/2510.10930/) (2025)
- [Alignment Makes Language Models Normative, Not Descriptive](https://mlsanity.com/paper/2603.17218/) (2026)
- [LLM Agents Display Human Biases but Exhibit Distinct Learning Patterns](https://mlsanity.com/paper/2503.10248/) (2025)
- [Would a Large Language Model Pay Extra for a View? Inferring Willingness to Pay from Subjective Choices](https://mlsanity.com/paper/2602.09802/) (2026)

## Abstract (arXiv)

In order for AI systems to communicate effectively with people, they must understand how we make decisions. However, people's decisions are not always rational, so the implicit internal models of human decision-making in Large Language Models (LLMs) must account for this. Previous empirical evidence seems to suggest that these implicit models are accurate -- LLMs offer believable proxies of human behavior, acting how we expect humans would in everyday interactions. However, by comparing LLM behavior and predictions to a large dataset of human decisions, we find that this is actually not the case: when both simulating and predicting people's choices, a suite of cutting-edge LLMs (GPT-4o & 4-Turbo, Llama-3-8B & 70B, Claude 3 Opus) assume that people are more rational than we really are. Specifically, these models deviate from human behavior and align more closely with a classic model of rational choice -- expected value theory. Interestingly, people also tend to assume that other people are rational when interpreting their behavior. As a consequence, when we compare the inferences that LLMs and people draw from the decisions of others using another psychological dataset, we find that these inferences are highly correlated. Thus, the implicit decision-making models of LLMs appear to be aligned with the human expectation that other people will act rationally, rather than with how people actually act.