mlsanityData to 14 Oct 2026

FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions

arXiv 2310.15421 ↗First posted 24 Oct 2023cs.CL cs.AIcited by 171

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Theory of mind (ToM) evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering. Our benchmark draws upon important theoretical requisites from psychology and necessary empirical considerations when evaluating large language models (LLMs). In particular, we formulate multiple types of questions that demand the same underlying reasoning to identify illusory or false sense of ToM capabilities in LLMs. We show that FANToM is challenging for state-of-the-art LLMs, which perform significantly worse than humans even with chain-of-thought reasoning or fine-tuning.

Key facts. Cited by 171 papers in the map (data through 14 Oct 2026). Builds on 23 of its 58 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

23 of 58 references.

References · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  9. 2021Ethical and social risks of harm from Language Modelscited by 966
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  11. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  12. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
11 more
  1. 2022UL2: Unifying Language Learning Paradigmscited by 295
  2. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  3. 2020Experience Grounds Languagecited by 242
  4. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  5. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  6. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  7. 2018Evaluating Theory of Mind in Question Answeringcited by 64
  8. 2023Boosting Theory-of-Mind Performance in Large Language Models via Promptingcited by 64
  9. 2018Modeling Naive Psychology of Characters in Simple Commonsense Storiescited by 60
  10. 2018What Makes Reading Comprehension Questions Easier?cited by 60
  11. 2022Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog Systemscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 39; 2025: 56; 2026: 75.

171 papers.

Sort:

By area:

  1. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  2. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  3. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  4. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 119
  5. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  6. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
  7. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 72
  8. 2024Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMscited by 59
  9. 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 57
  10. 2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Modelscited by 54
  11. 2024EmoBench: Evaluating the Emotional Intelligence of Large Language Modelscited by 53
  12. 2024NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surroundingcited by 51
38 more
  1. 2024MuMA-ToM: Multi-modal Multi-Agent Theory of Mindcited by 51
  2. 2024BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Datacited by 50
  3. 2024Beyond Preferences in AI Alignmentcited by 49
  4. 2025Language Models use Lookbacks to Track Beliefscited by 39
  5. 2025The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMscited by 36
  6. 2024Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extractioncited by 33
  7. 2024Position: Theory of Mind Benchmarks are Broken for Large Language Modelscited by 31
  8. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 30
  9. 2025Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Modelscited by 29
  10. 2025Theory of Mind in Large Language Models: Assessment and Enhancementcited by 25
  11. 2025Does It Make Sense to Speak of Introspection in Large Language Models?cited by 25
  12. 2025AutoToM: Scaling Model-based Mental Inference via Automated Agent Modelingcited by 23
  13. 2025Re-evaluating Theory of Mind evaluation in large language modelscited by 23
  14. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 22
  15. 2024TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Modelscited by 22
  16. 2024Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Modelscited by 22
  17. 2024TimeToM: Temporal Space is the Key to Unlocking the Door of Large Language Models' Theory-of-Mindcited by 21
  18. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  19. 2026Large Language Model Reasoning Failurescited by 21
  20. 2025Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human Statescited by 19
  21. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  22. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 18
  23. 2025Decompose-ToM: Enhancing Theory of Mind Reasoning in Large Language Models through Simulation and Task Decompositioncited by 17
  24. 2025PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialoguescited by 17
  25. 2026GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversationscited by 17
  26. 2024A Notion of Complexity for Theory of Mind via Discrete World Modelscited by 16
  27. 2025SoMi-ToM: Evaluating Multi-Perspective Theory of Mind in Embodied Social Interactionscited by 16
  28. 2025Social World Modelscited by 16
  29. 2024Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Modelscited by 15
  30. 2025NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learningcited by 15
  31. 2024From Imitation to Introspection: Probing Self-Consciousness in Language Modelscited by 14
  32. 2025Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspectivecited by 14
  33. 2024Poser: Unmasking Alignment Faking LLMs by Manipulating Their Internalscited by 13
  34. 2025Large Language Models as Theory of Mind Aware Generative Agents with Counterfactual Reflectioncited by 13
  35. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  36. 2026OdysSim: Building Foundation Models for Human Behavior Simulationcited by 13
  37. 2024Boundless Socratic Learning with Language Gamescited by 12
  38. 2025ToMAP: Training Opponent-Aware LLM Persuaders with Theory of Mindcited by 12

50 most cited of 171 All 171 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Understanding Social Reasoning in Language Models with Language Models
  2. 2023HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
  3. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities
  4. 2024Probing the Robustness of Theory of Mind in Large Language Models
  5. 2026Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action
  6. 2024Do LLMs Exhibit Human-Like Reasoning? Evaluating Theory of Mind in LLMs for Open-Ended Responses
  7. 2023MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
  8. 2026Beyond Sally-Anne: Evaluating Theory of Mind in LLMs using Epistemic Schelling Points