# ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind

arXiv: https://arxiv.org/abs/2501.08838 · HTML: https://mlsanity.com/paper/2501.08838/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jan 2025.

## Built on by (most cited first, 21 of 21)

- [SIV-Bench: A Video Benchmark for Social Interaction Understanding and Reasoning](https://mlsanity.com/paper/2506.05425/) (2025)
- [PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialogues](https://mlsanity.com/paper/2502.21017/) (2025)
- [GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations](https://mlsanity.com/paper/2605.14498/) (2026)
- [OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence](https://mlsanity.com/paper/2602.08683/) (2026)
- [TactfulToM: Do LLMs Have the Theory of Mind Ability to Understand White Lies?](https://mlsanity.com/paper/2509.17054/) (2025)
- [Let's Put Ourselves in Sally's Shoes: Shoes-of-Others Prefilling Improves Theory of Mind in Large Language Models](https://mlsanity.com/paper/2506.05970/) (2025)
- [To Think or Not To Think, That is The Question for Large Reasoning Models in Theory of Mind Tasks](https://mlsanity.com/paper/2602.10625/) (2026)
- [DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories](https://mlsanity.com/paper/2604.20443/) (2026)
- [From Shortcuts to Reasoning: Robust Post-Training of Theory of Mind with Reinforcement Learning](https://mlsanity.com/paper/2606.09092/) (2026)
- [MafiaScope: Non-Invasive, Time-Resolved Belief Probing for LLM Agents in Social Deduction Games](https://mlsanity.com/paper/2607.10645/) (2026)
- [Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluations](https://mlsanity.com/paper/2605.15205/) (2026)
- [MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings](https://mlsanity.com/paper/2607.19235/) (2026)
- [Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States](https://mlsanity.com/paper/2609.15972/) (2026)
- [TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence](https://mlsanity.com/paper/2505.24500/) (2025)
- [SocialNLI: A Dialogue-Centric Social Inference Dataset](https://mlsanity.com/paper/2510.05458/) (2025)
- [CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs](https://mlsanity.com/paper/2603.11915/) (2026)
- [Dynamic Theory of Mind as a Temporal Memory Problem: Evidence from Large Language Models](https://mlsanity.com/paper/2603.14646/) (2026)
- [Avalon-ToM-Bench: Evaluating Fine-Grained Theory of Mind via Asymmetric Game Mechanics](https://mlsanity.com/paper/2608.09638/) (2026)
- [Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking](https://mlsanity.com/paper/2608.25654/) (2026)
- [Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop](https://mlsanity.com/paper/2610.15692/) (2026)
- [Jev-ToM: Evaluating Theory of Mind in Typed Decision Models](https://mlsanity.com/paper/2610.18610/) (2026)

## Abstract (arXiv)

Existing Theory of Mind (ToM) benchmarks diverge from real-world scenarios in three aspects: 1) they assess a limited range of mental states such as beliefs, 2) false beliefs are not comprehensively explored, and 3) the diverse personality traits of characters are overlooked. To address these challenges, we introduce ToMATO, a new ToM benchmark formulated as multiple-choice QA over conversations. ToMATO is generated via LLM-LLM conversations featuring information asymmetry. By employing a prompting method that requires role-playing LLMs to verbalize their thoughts before each utterance, we capture both first- and second-order mental states across five categories: belief, intention, desire, emotion, and knowledge. These verbalized thoughts serve as answers to questions designed to assess the mental states of characters within conversations. Furthermore, the information asymmetry introduced by hiding thoughts from others induces the generation of false beliefs about various mental states. Assigning distinct personality traits to LLMs further diversifies both utterances and thoughts. ToMATO consists of 5.4k questions, 753 conversations, and 15 personality trait patterns. Our analysis shows that this dataset construction approach frequently generates false beliefs due to the information asymmetry between role-playing LLMs, and effectively reflects diverse personalities. We evaluate nine LLMs on ToMATO and find that even GPT-4o mini lags behind human performance, especially in understanding false beliefs, and lacks robustness to various personality traits.