# Evaluating Theory of Mind in Question Answering

arXiv: https://arxiv.org/abs/1808.09352 · HTML: https://mlsanity.com/paper/1808.09352/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2018.

## Built on by (most cited first, 30 of 64)

- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://mlsanity.com/paper/2306.00924/) (2023)
- [The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models](https://mlsanity.com/paper/2406.05761/) (2024)
- [MMToM-QA: Multimodal Theory of Mind Question Answering](https://mlsanity.com/paper/2401.08743/) (2024)
- [How FaR Are Large Language Models From Agents with Theory-of-Mind?](https://mlsanity.com/paper/2310.03051/) (2023)
- [A fine-grained comparison of pragmatic language understanding in humans and language models](https://mlsanity.com/paper/2212.06801/) (2022)
- [HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models](https://mlsanity.com/paper/2310.16755/) (2023)
- [SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs](https://mlsanity.com/paper/2410.13648/) (2024)
- [OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2402.06044/) (2024)
- [NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding](https://mlsanity.com/paper/2404.13627/) (2024)
- [Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Tests](https://mlsanity.com/paper/2310.20320/) (2023)
- [Few-shot Language Coordination by Modeling Theory of Mind](https://mlsanity.com/paper/2107.05697/) (2021)
- [Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models](https://mlsanity.com/paper/2310.19619/) (2023)
- [Language Models Represent Beliefs of Self and Others](https://mlsanity.com/paper/2402.18496/) (2024)
- [ToMChallenges: A Principle-Guided Dataset and Diverse Evaluation Tasks for Exploring Theory of Mind](https://mlsanity.com/paper/2305.15068/) (2023)
- [InterIntent: Investigating Social Intelligence of LLMs via Intention Understanding in an Interactive Game Context](https://mlsanity.com/paper/2406.12203/) (2024)
- [Theory of Mind in Large Language Models: Assessment and Enhancement](https://mlsanity.com/paper/2505.00026/) (2025)
- [Cognitive science as a source of forward and inverse models of human decisions for robotics and control](https://mlsanity.com/paper/2109.00127/) (2021)
- [I Cast Detect Thoughts: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons](https://mlsanity.com/paper/2212.10060/) (2022)
- [I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench](https://mlsanity.com/paper/2401.17882/) (2024)
- [TimeToM: Temporal Space is the Key to Unlocking the Door of Large Language Models' Theory-of-Mind](https://mlsanity.com/paper/2407.01455/) (2024)
- [Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models](https://mlsanity.com/paper/2409.20222/) (2024)
- [Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind](https://mlsanity.com/paper/2211.04684/) (2022)
- [Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human States](https://mlsanity.com/paper/2505.17663/) (2025)

## Abstract (arXiv)

We propose a new dataset for evaluating question answering models with respect to their capacity to reason about beliefs. Our tasks are inspired by theory-of-mind experiments that examine whether children are able to reason about the beliefs of others, in particular when those beliefs differ from reality. We evaluate a number of recent neural models with memory augmentation. We find that all fail on our tasks, which require keeping track of inconsistent states of the world; moreover, the models' accuracy decreases notably when random sentences are introduced to the tasks at test.