mlsanityData to 7 Oct 2026

Understanding Social Reasoning in Language Models with Language Models

arXiv 2306.15448 ↗First posted 21 Jun 2023cs.CL cs.AI cs.HCcited by 166

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

As Large Language Models (LLMs) become increasingly integrated into our everyday lives, understanding their ability to comprehend human mental states becomes critical for ensuring effective interactions. However, despite the recent attempts to assess the Theory-of-Mind (ToM) reasoning capabilities of LLMs, the degree to which these models can align with human ToM remains a nuanced topic of exploration. This is primarily due to two distinct challenges: (1) the presence of inconsistent results from previous evaluations, and (2) concerns surrounding the validity of existing evaluation methodologies. To address these challenges, we present a novel framework for procedurally generating evaluations with LLMs by populating causal templates. Using our framework, we create a new social reasoning benchmark (BigToM) for LLMs which consists of 25 controls and 5,000 model-written evaluations. We find that human participants rate the quality of our benchmark higher than previous crowd-sourced evaluations and comparable to expert-written evaluations. Using BigToM, we evaluate the social reasoning capabilities of a variety of LLMs and compare model performances with human performance. Our results suggest that GPT4 has ToM capabilities that mirror human inference patterns, though less reliable, while other LLMs struggle.

Key facts. Cited by 166 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 50 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

24 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models24 May
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  3. 2022Language models show human-like content effects on reasoning tasks14 Jul

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  8. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  9. 2016Cooperative Inverse Reinforcement Learningcited by 298
  10. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  11. 2018Machine Theory of Mindcited by 222
  12. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
9 more
  1. 2021Generating Datasets with Pretrained Language Modelscited by 192
  2. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  3. 2018Modeling Others using Oneself in Multi-Agent Reinforcement Learningcited by 64
  4. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
  5. 2022Do Large Language Models know what humans know?cited by 60
  6. 2021AGENT: A Benchmark for Core Psychological Reasoningcited by 42
  7. 2021Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of otherscited by 39
  8. 2023ToMChallenges: A Principle-Guided Dataset and Diverse Evaluation Tasks for Exploring Theory of Mindcited by 28
  9. 2022Learning Causal Overhypotheses through Exploration in Children and Computational Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 51; 2025: 58; 2026: 51.

166 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  3. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
  4. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  5. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  6. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  7. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 86
  8. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  9. 2024Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Datacited by 65
  10. 2024LLMs achieve adult human performance on higher-order theory of mind taskscited by 59
  11. 2024Are Large Language Models Consistent over Value-laden Questions?cited by 58
  12. 2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Modelscited by 54
38 more
  1. 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
  2. 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 51
  3. 2024NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surroundingcited by 49
  4. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  5. 2024Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Modelscited by 45
  6. 2023Visual cognition in multimodal large language modelscited by 38
  7. 2025Language Models use Lookbacks to Track Beliefscited by 36
  8. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  9. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30
  10. 2024Language Models Represent Beliefs of Self and Otherscited by 29
  11. 2025Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Modelscited by 28
  12. 2024LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Outputcited by 27
  13. 2024InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Contextcited by 25
  14. 2025On Benchmarking Human-Like Intelligence in Machinescited by 24
  15. 2025Theory of Mind in Large Language Models: Assessment and Enhancementcited by 24
  16. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  17. 2025AutoToM: Scaling Model-based Mental Inference via Automated Agent Modelingcited by 23
  18. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
  19. 2024Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Modelscited by 22
  20. 2024TimeToM: Temporal Space is the Key to Unlocking the Door of Large Language Models' Theory-of-Mindcited by 20
  21. 2025Population-Aligned Persona Generation for LLM-based Social Simulationcited by 20
  22. 2025Decompose-ToM: Enhancing Theory of Mind Reasoning in Large Language Models through Simulation and Task Decompositioncited by 17
  23. 2025Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human Statescited by 17
  24. 2025PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialoguescited by 16
  25. 2025SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactionscited by 16
  26. 2024See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknessescited by 15
  27. 2024Large Language Models Fall Short: Understanding Complex Relationships in Detective Narrativescited by 14
  28. 2024M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmarkcited by 14
  29. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  30. 2024Human-like Affective Cognition in Foundation Modelscited by 13
  31. 2024CausalEval: Towards Better Causal Reasoning in Language Modelscited by 13
  32. 2024A Survey on Human-Centric LLMscited by 13
  33. 2024The Call for Socially Aware Language Technologiescited by 12
  34. 2024Imagining and building wise machines: The centrality of AI metacognitioncited by 12
  35. 2025From Perceptions to Decisions: Wildfire Evacuation Decision Prediction with Behavioral Theory-informed LLMscited by 12
  36. 2025WinClick: GUI Grounding with Multimodal Large Language Modelscited by 11
  37. 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Modelscited by 11
  38. 2024EgoSocialArena: Benchmarking the Social Intelligence of Large Language Models from a First-person Perspectivecited by 10

50 most cited of 166 All 166 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Probing the Robustness of Theory of Mind in Large Language Models
  2. 2024Multi-ToM: Evaluating Multilingual Theory of Mind Capabilities in Large Language Models
  3. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  4. 2024ToM-LM: Delegating Theory of Mind Reasoning to External Symbolic Executors in Large Language Models
  5. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
  6. 2024Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning
  7. 2026Traces of Social Competence in Large Language Models
  8. 2023Evaluating Large Language Models in Theory of Mind Tasks