# From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2506.14224 · HTML: https://mlsanity.com/paper/2506.14224/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 44 references that are in the map.
- First posted on arXiv 17 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 18 of 18)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023)
- [Evaluating Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2302.02083/) (2023)
- [How FaR Are Large Language Models From Agents with Theory-of-Mind?](https://mlsanity.com/paper/2310.03051/) (2023)
- [OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2402.06044/) (2024)
- [Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Tests](https://mlsanity.com/paper/2310.20320/) (2023)
- [Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information](https://mlsanity.com/paper/2408.02559/) (2024)
- [Theory of Mind as Intrinsic Motivation for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.01158/) (2023)
- [Who is Mistaken?](https://mlsanity.com/paper/1612.01175/) (2016)
- [Through the Theory of Mind's Eye: Reading Minds with Multimodal Video Large Language Models](https://mlsanity.com/paper/2406.13763/) (2024)
- [Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models](https://mlsanity.com/paper/2408.05241/) (2024)

## Built on by (most cited first, 5 of 5)

- [MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agents](https://mlsanity.com/paper/2511.23055/) (2025)
- [Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?](https://mlsanity.com/paper/2512.17394/) (2025)
- [Modeling the Mental World for Embodied AI: A Comprehensive Review](https://mlsanity.com/paper/2601.02378/) (2025)
- [Video-Only ToM: Enhancing Theory of Mind in Multimodal Large Language Models](https://mlsanity.com/paper/2603.24484/) (2026)
- [MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention](https://mlsanity.com/paper/2606.01063/) (2026)

## Abstract (arXiv)

As large language models evolve, there is growing anticipation that they will emulate human-like Theory of Mind (ToM) to assist with routine tasks. However, existing methods for evaluating machine ToM focus primarily on unimodal models and largely treat these models as black boxes, lacking an interpretative exploration of their internal mechanisms. In response, this study adopts an approach based on internal mechanisms to provide an interpretability-driven assessment of ToM in multimodal large language models (MLLMs). Specifically, we first construct a multimodal ToM test dataset, GridToM, which incorporates diverse belief testing tasks and perceptual information from multiple perspectives. Next, our analysis shows that attention heads in multimodal large models can distinguish cognitive information across perspectives, providing evidence of ToM capabilities. Furthermore, we present a lightweight, training-free approach that significantly enhances the model's exhibited ToM by adjusting in the direction of the attention head.