# The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters

arXiv: https://arxiv.org/abs/2501.01705 · HTML: https://mlsanity.com/paper/2501.01705/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 3 Jan 2025.

## Builds on (most cited first, 18 of 18)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [Understanding Social Reasoning in Language Models with Language Models](https://mlsanity.com/paper/2306.15448/) (2023)
- [FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions](https://mlsanity.com/paper/2310.15421/) (2023)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension](https://mlsanity.com/paper/2203.13947/) (2022)
- [Fine-Grained Spoiler Detection from Large-Scale Review Corpora](https://mlsanity.com/paper/1905.13416/) (2019)
- [Evaluating Theory of Mind in Question Answering](https://mlsanity.com/paper/1808.09352/) (2018)
- [NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding](https://mlsanity.com/paper/2404.13627/) (2024)
- [ToMChallenges: A Principle-Guided Dataset and Diverse Evaluation Tasks for Exploring Theory of Mind](https://mlsanity.com/paper/2305.15068/) (2023)
- [COKE: A Cognitive Knowledge Graph for Machine Theory of Mind](https://mlsanity.com/paper/2305.05390/) (2023)
- [Personality Understanding of Fictional Characters during Book Reading](https://mlsanity.com/paper/2305.10156/) (2023)
- [Few-Shot Character Understanding in Movies as an Assessment to Meta-Learning of Theory-of-Mind](https://mlsanity.com/paper/2211.04684/) (2022)
- [RELIC: Retrieving Evidence for Literary Claims](https://mlsanity.com/paper/2203.10053/) (2022)
- [Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions](https://mlsanity.com/paper/2402.13551/) (2024)

## Built on by (most cited first, 8 of 8)

- [SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension](https://mlsanity.com/paper/2508.01959/) (2025)
- [PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts](https://mlsanity.com/paper/2508.09848/) (2025)
- [Query-focused and Memory-aware Reranker for Long Context Processing](https://mlsanity.com/paper/2602.12192/) (2026)
- [Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey](https://mlsanity.com/paper/2602.15851/) (2026)
- [HGMEM: Hypergraph-based Working Memory to Improve Multi-step RAG for Long-Context Complex Relational Modeling](https://mlsanity.com/paper/2512.23959/) (2025)
- [MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.00610/) (2026)
- [Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answers](https://mlsanity.com/paper/2604.11246/) (2026)
- [MindZero: Learning Online Mental Reasoning With Zero Annotations](https://mlsanity.com/paper/2606.00240/) (2026)

## Abstract (arXiv)

Theory-of-Mind (ToM) is a fundamental psychological capability that allows humans to understand and interpret the mental states of others. Humans infer others' thoughts by integrating causal cues and indirect clues from broad contextual information, often derived from past interactions. In other words, human ToM heavily relies on the understanding about the backgrounds and life stories of others. Unfortunately, this aspect is largely overlooked in existing benchmarks for evaluating machines' ToM capabilities, due to their usage of short narratives without global context, especially personal background of characters. In this paper, we verify the importance of comprehensive contextual understanding about personal backgrounds in ToM and assess the performance of LLMs in such complex scenarios. To achieve this, we introduce CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels. Our human study reveals a significant disparity in performance: the same group of educated participants performs dramatically better when they have read the novels compared to when they have not. In parallel, our experiments on state-of-the-art LLMs, including the very recent o1 and DeepSeek-R1 models, show that LLMs still perform notably worse than humans, despite that they have seen these stories during pre-training. This highlights the limitations of current LLMs in capturing the nuanced contextual information required for ToM reasoning.