mlsanityData to 7 Oct 2026

Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks

arXiv 2302.08399 ↗First posted 16 Feb 2023cs.AI cs.CLcited by 288

Not on the map. Why?

Cited by 62 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Intuitive psychology is a pillar of common-sense reasoning. The replication of this reasoning in machine intelligence is an important stepping-stone on the way to human-like artificial intelligence. Several recent tasks and benchmarks for examining this reasoning in Large-Large Models have focused in particular on belief attribution in Theory-of-Mind tasks. These tasks have shown both successes and failures. We consider in particular a recent purported success case, and show that small variations that maintain the principles of ToM turn the results on their head. We argue that in general, the zero-hypothesis for model evaluation in intuitive psychology should be skeptical, and that outlying failure cases should outweigh average success rates. We also consider what possible future successes on Theory-of-Mind tasks by more powerful LLMs would mean for ToM tasks with people.

Key facts. Cited by 288 papers in the map (data through 7 Oct 2026). Builds on 11 of its 39 references that are in the map. First posted on arXiv 16 Feb 2023.

Builds on

11 of 39 references.

References · 11

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2018Machine Theory of Mindcited by 222
  7. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  8. 2018Evaluating Theory of Mind in Question Answeringcited by 64
  9. 2022Testing Relational Understanding in Text-Guided Image Generationcited by 59
  10. 2021AGENT: A Benchmark for Core Psychological Reasoningcited by 42
  11. 2021Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of otherscited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 57; 2024: 72; 2025: 80; 2026: 79.

288 papers.

Sort:

By area:

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  3. 2023Playing repeated games with Large Language Modelscited by 213
  4. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  5. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
  6. 2023Personality Traits in Large Language Modelscited by 166
  7. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  8. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  9. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  10. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  11. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Modelscited by 129
  12. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
38 more
  1. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  2. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  3. 2024Emergent social conventions and collective bias in LLM populationscited by 115
  4. 2025Agentic Large Language Models, a surveycited by 105
  5. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  6. 2023Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiationcited by 94
  7. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  8. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 86
  9. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  10. 2023Evaluating Cognitive Maps and Planning in Large Language Models with CogEvalcited by 77
  11. 2023How FaR Are Large Language Models From Agents with Theory-of-Mind?cited by 75
  12. 2023Eight Things to Know about Large Language Modelscited by 73
  13. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  14. 2024Building Machines that Learn and Think with Peoplecited by 73
  15. 2023Efficient Benchmarking of Language Modelscited by 71
  16. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
  17. 2023Turning large language models into cognitive modelscited by 66
  18. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
  19. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Modelscited by 61
  20. 2024CogBench: a large language model walks into a psychology labcited by 60
  21. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  22. 2024LLMs achieve adult human performance on higher-order theory of mind taskscited by 59
  23. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  24. 2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Modelscited by 54
  25. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  26. 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
  27. 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 51
  28. 2024NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surroundingcited by 49
  29. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  30. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 48
  31. 2025Why human-AI relationships need socioaffective alignmentcited by 47
  32. 2023Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Testscited by 46
  33. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46
  34. 2023Prompting is not a substitute for probability measurements in large language modelscited by 43
  35. 2024Werewolf Arena: A Case Study in LLM Evaluation via Social Deductioncited by 42
  36. 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41
  37. 2023Visual cognition in multimodal large language modelscited by 38
  38. 2025Language Models use Lookbacks to Track Beliefscited by 36

50 most cited of 288 All 288 (JSON)