mlsanityData to 7 Oct 2026

Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs

arXiv 2210.13312 ↗First posted 24 Oct 2022cs.CL cs.AIcited by 220

On the map: World models → World models for language agents

Abstract (arXiv)

Social intelligence and Theory of Mind (ToM), i.e., the ability to reason about the different mental states, intents, and reactions of all people involved, allow humans to effectively navigate and understand everyday social interactions. As NLP systems are used in increasingly complex social situations, their ability to grasp social dynamics becomes crucial. In this work, we examine the open question of social intelligence and Theory of Mind in modern NLP systems from an empirical and theory-based perspective. We show that one of today's largest language models (GPT-3; Brown et al., 2020) lacks this kind of social intelligence out-of-the box, using two tasks: SocialIQa (Sap et al., 2019), which measures models' ability to understand intents and reactions of participants of social interactions, and ToMi (Le et al., 2019), which measures whether models can infer mental states and realities of participants of situations. Our results show that models struggle substantially at these Theory of Mind tasks, with well-below-human accuracies of 55% and 60% on SocialIQa and ToMi, respectively. To conclude, we draw on theories from pragmatics to contextualize this shortcoming of large language models, by examining the limitations stemming from their data, neural architecture, and training paradigms. Challenging the prevalent narrative that only scale is needed, we posit that person-centric NLP approaches might be more effective towards neural Theory of Mind. In our updated version, we also analyze newer instruction tuned and RLFH models for neural ToM. We find that even ChatGPT and GPT-4 do not display emergent Theory of Mind; strikingly even GPT-4 performs only 60% accuracy on the ToMi questions related to mental states and realities.

Key facts. Cited by 220 papers in the map (data through 7 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 52 of its 138 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

52 of 138 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Language Models are Few-Shot Learners28 May

Other references · 51

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  12. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
38 more
  1. 2020Autoregressive Entity Retrievalcited by 333
  2. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  3. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  4. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  5. 2020Experience Grounds Languagecited by 241
  6. 2019Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Goodcited by 207
  7. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  8. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  9. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  10. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  11. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  12. 2020GLUCOSE: GeneraLized and COntextualized Story Explanationscited by 85
  13. 2021Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approachcited by 75
  14. 2020Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learningcited by 65
  15. 2018Evaluating Theory of Mind in Question Answeringcited by 64
  16. 2021MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Taskscited by 62
  17. 2018Modeling Naive Psychology of Characters in Simple Commonsense Storiescited by 59
  18. 2021ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mindcited by 57
  19. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  20. 2020Knowledge-Aware Language Model Pretrainingcited by 52
  21. 2016Tracking the World State with Recurrent Entity Networkscited by 51
  22. 2021What Context Features Can Transformer Language Models Use?cited by 49
  23. 2022Training Language Models with Language Feedbackcited by 48
  24. 2020Pre-training Text-to-Text Transformers for Concept-centric Common Sensecited by 42
  25. 2021Emergent Communication of Generalizationscited by 41
  26. 2017The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Taskcited by 39
  27. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  28. 2021Few-shot Language Coordination by Modeling Theory of Mindcited by 32
  29. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
  30. 2019Continual adaptation for efficient machine communicationcited by 28
  31. 2022When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to itcited by 22
  32. 2017Are distributional representations ready for the real world? Evaluating word vectors for grounded perceptual meaningcited by 19
  33. 2017Dynamic Entity Representations in Neural Language Modelscited by 19
  34. 2018Machine Common Sense Concept Papercited by 17
  35. 2021Situated Dialogue Learning through Procedural Environment Generationcited by 9
  36. 2016Reference-Aware Language Modelscited by 8
  37. 2019The Referential Reader: A Recurrent Entity Network for Anaphora Resolutioncited by 8
  38. 2021IoT-Enabled Social Relationships Meet Artificial Social Intelligencecited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 61; 2024: 68; 2025: 42; 2026: 43.

220 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  3. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  4. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  5. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  6. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  7. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  8. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  9. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 217
  10. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  11. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
  12. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
38 more
  1. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 161
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  3. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  4. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 129
  5. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Modelscited by 129
  6. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  7. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  8. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  9. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  10. 2023Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiationcited by 94
  11. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  12. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  13. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 86
  14. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85
  15. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  16. 2023How FaR Are Large Language Models From Agents with Theory-of-Mind?cited by 75
  17. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
  18. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  19. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
  20. 2023Strategic Reasoning with Language Modelscited by 64
  21. 2023Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?cited by 62
  22. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Modelscited by 61
  23. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  24. 2024LLMs achieve adult human performance on higher-order theory of mind taskscited by 59
  25. 2023Emotional Intelligence of Large Language Modelscited by 58
  26. 2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Modelscited by 54
  27. 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 51
  28. 2024NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surroundingcited by 49
  29. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  30. 2023You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instrumentscited by 48
  31. 2023Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Testscited by 46
  32. 2023Merlin:Empowering Multimodal LLMs with Foresight Mindscited by 46
  33. 2023Entity Tracking in Language Modelscited by 44
  34. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thoughtcited by 41
  35. 2024LLM Harmony: Multi-Agent Communication for Problem Solvingcited by 34
  36. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talkcited by 34
  37. 2024The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Unitscited by 34
  38. 2023Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writerscited by 32

50 most cited of 220 All 220 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Probing the Robustness of Theory of Mind in Large Language Models
  2. 2024Language Models Represent Beliefs of Self and Others
  3. 2024Multi-ToM: Evaluating Multilingual Theory of Mind Capabilities in Large Language Models
  4. 2025Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
  5. 2025Infusing Theory of Mind into Socially Intelligent LLM Agents
  6. 2024Position: Theory of Mind Benchmarks are Broken for Large Language Models
  7. 2026On Emergent Social World Models -- Evidence for Functional Integration of Theory of Mind and Pragmatic Reasoning in Language Models
  8. 2025Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models