Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Research on mental state reasoning in language models (LMs) has the potential to inform theories of human social cognition--such as the theory that mental state reasoning emerges in part from language exposure--and our understanding of LMs themselves. Yet much published work on LMs relies on a relatively small sample of closed-source LMs, limiting our ability to rigorously test psychological theories and evaluate LM capacities. Here, we replicate and extend published work on the false belief task by assessing LM mental state reasoning behavior across 41 open-weight models (from distinct model families). We find sensitivity to implied knowledge states in 34% of the LMs tested; however, consistent with prior work, none fully ``explain away'' the effect in humans. Larger LMs show increased sensitivity and also exhibit higher psychometric predictive power. Finally, we use LM behavior to generate and test a novel hypothesis about human cognition: both humans and LMs show a bias towards attributing false beliefs when knowledge states are cued using a non-factive verb (``John thinks...'') than when cued indirectly (``John looks in the...''). Unlike the primary effect of knowledge states, where human sensitivity exceeds that of LMs, the magnitude of the human knowledge cue effect falls squarely within the distribution of LM effect sizes-suggesting that distributional statistics of language can in principle account for the latter but not the former in humans. These results demonstrate the value of using larger samples of open-weight LMs to test theories of human cognition and evaluate LM capacities.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 15 of its 73 references that are in the map. First posted on arXiv 17 Feb 2026.
Builds on
15 of 73 references.
References · 15
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 20242 OLMo 2 Furiouscited by 302
- 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
- 2023Evaluating Large Language Models in Theory of Mind Taskscited by 217
- 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
- 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 51
- 2024PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarkscited by 18
- 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models
- 2023A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models
- 2023Understanding Social Reasoning in Language Models with Language Models
- 2024Language Models Represent Beliefs of Self and Others
- 2024Belief in the Machine: Investigating Epistemological Blind Spots of Language Models
- 2026Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models
- 2025XToM: Exploring the Multilingual Theory of Mind for Large Language Models
- 2026Beliefs and Behavior in Language Models