# Evaluating and Modeling Social Intelligence: A Comparative Study of Human and AI Capabilities

arXiv: https://arxiv.org/abs/2405.11841 · HTML: https://mlsanity.com/paper/2405.11841/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 20 May 2024.

## Builds on (most cited first, 17 of 17)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [The Debate Over Understanding in AI's Large Language Models](https://mlsanity.com/paper/2210.13966/) (2022)
- [Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks](https://mlsanity.com/paper/2205.05718/) (2022)
- [GPT-4 Can't Reason](https://mlsanity.com/paper/2308.03762/) (2023)
- [ToMChallenges: A Principle-Guided Dataset and Diverse Evaluation Tasks for Exploring Theory of Mind](https://mlsanity.com/paper/2305.15068/) (2023)
- [A mathematical theory of cooperative communication](https://mlsanity.com/paper/1910.02822/) (2019)
- [Learning Triadic Belief Dynamics in Nonverbal Communication from Videos](https://mlsanity.com/paper/2104.02841/) (2021)
- [Optimal Cooperative Inference](https://mlsanity.com/paper/1705.08971/) (2017)
- [Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models](https://mlsanity.com/paper/2307.03762/) (2023)
- [Acting as Inverse Inverse Planning](https://mlsanity.com/paper/2305.16913/) (2023)

## Built on by (most cited first, 6 of 6)

- [R^3-VQA: "Read the Room" by Video Social Reasoning](https://mlsanity.com/paper/2505.04147/) (2025)
- [The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoning](https://mlsanity.com/paper/2510.04141/) (2025)
- [LLM Agents Display Human Biases but Exhibit Distinct Learning Patterns](https://mlsanity.com/paper/2503.10248/) (2025)
- [InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes under Herd Behavior](https://mlsanity.com/paper/2507.06528/) (2025)
- [AREG: Adversarial Resource Extraction Game for Evaluating Persuasion and Resistance in Large Language Models](https://mlsanity.com/paper/2602.16639/) (2026)
- [Human-Centered Human-AI Collaboration (HCHAC)](https://mlsanity.com/paper/2505.22477/) (2025)

## Abstract (arXiv)

Facing the current debate on whether Large Language Models (LLMs) attain near-human intelligence levels (Mitchell & Krakauer, 2023; Bubeck et al., 2023; Kosinski, 2023; Shiffrin & Mitchell, 2023; Ullman, 2023), the current study introduces a benchmark for evaluating social intelligence, one of the most distinctive aspects of human cognition. We developed a comprehensive theoretical framework for social dynamics and introduced two evaluation tasks: Inverse Reasoning (IR) and Inverse Inverse Planning (IIP). Our approach also encompassed a computational model based on recursive Bayesian inference, adept at elucidating diverse human behavioral patterns. Extensive experiments and detailed analyses revealed that humans surpassed the latest GPT models in overall performance, zero-shot learning, one-shot generalization, and adaptability to multi-modalities. Notably, GPT models demonstrated social intelligence only at the most basic order (order = 0), in stark contrast to human social intelligence (order >= 2). Further examination indicated a propensity of LLMs to rely on pattern recognition for shortcuts, casting doubt on their possession of authentic human-level social intelligence. Our codes, dataset, appendix and human data are released at https://github.com/bigai-ai/Evaluate-n-Model-Social-Intelligence.