# Mutual Theory of Mind for Human-AI Communication

arXiv: https://arxiv.org/abs/2210.03842 · HTML: https://mlsanity.com/paper/2210.03842/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 62 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 7 of 7)

- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Expanding Explainability: Towards Social Transparency in AI systems](https://mlsanity.com/paper/2101.04719/) (2021)
- [Human-centered Explainable AI: Towards a Reflective Sociotechnical Approach](https://mlsanity.com/paper/2002.01092/) (2020)
- [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019)
- [Perfection Not Required? Human-AI Partnerships in Code Translation](https://mlsanity.com/paper/2104.03820/) (2021)
- [Human-Centered Artificial Intelligence and Machine Learning](https://mlsanity.com/paper/1901.11184/) (2019)
- [What do they know about me? Contents and Concerns of Online Behavioral Profiles](https://mlsanity.com/paper/1506.01675/) (2015)

## Built on by (most cited first, 15 of 15)

- [Position: Theory of Mind Benchmarks are Broken for Large Language Models](https://mlsanity.com/paper/2412.19726/) (2024)
- [Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmap](https://mlsanity.com/paper/2410.06107/) (2024)
- [Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Task](https://mlsanity.com/paper/2409.08811/) (2024)
- [Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective](https://mlsanity.com/paper/2504.10839/) (2025)
- [Toward Metaphor-Fluid Conversation Design for Voice User Interfaces](https://mlsanity.com/paper/2502.11554/) (2025)
- [Scopes of Alignment](https://mlsanity.com/paper/2501.12405/) (2025)
- [Knowledge Affordances for Hybrid Human-AI Information Seeking](https://mlsanity.com/paper/2604.27539/) (2026)
- [Towards Conversational Development Environments: Using Theory-of-Mind and Multi-Agent Architectures for Requirements Refinement](https://mlsanity.com/paper/2505.20973/) (2025)
- [Conversational Alignment with Artificial Intelligence in Context](https://mlsanity.com/paper/2505.22907/) (2025)
- [Visual and Cognitive Demands of a Large Language Model-Powered In-vehicle Conversational Agent](https://mlsanity.com/paper/2601.15034/) (2026)
- [Situated, Dynamic, and Subjective: Envisioning the Design of Theory-of-Mind-Enabled Everyday AI with Industry Practitioners](https://mlsanity.com/paper/2602.11342/) (2026)
- [When Researchers Say Mental Model/Theory of Mind of AI, What Are They Really Talking About?](https://mlsanity.com/paper/2510.02660/) (2025)
- [What Do You Mean? Exploring How Humans and AI Interact with Symbols and Meanings in Their Interactions](https://mlsanity.com/paper/2510.05378/) (2025)
- [Mixed-Initiative Context: Structuring and Managing Context for Human-AI Collaboration](https://mlsanity.com/paper/2604.07121/) (2026)
- [Towards Cognitive Process-Aware Proactive Writing Support](https://mlsanity.com/paper/2608.30424/) (2026)

## Abstract (arXiv)

New developments are enabling AI systems to perceive, recognize, and respond with social cues based on inferences made from humans' explicit or implicit behavioral and verbal cues. These AI systems, equipped with an equivalent of human's Theory of Mind (ToM) capability, are currently serving as matchmakers on dating platforms, assisting student learning as teaching assistants, and enhancing productivity as work partners. They mark a new era in human-AI interaction (HAI) that diverges from traditional human-computer interaction (HCI), where computers are commonly seen as tools instead of social actors. Designing and understanding the human perceptions and experiences in this emerging HAI era becomes an urgent and critical issue for AI systems to fulfill human needs and mitigate risks across social contexts. In this paper, we posit the Mutual Theory of Mind (MToM) framework, inspired by our capability of ToM in human-human communications, to guide this new generation of HAI research by highlighting the iterative and mutual shaping nature of human-AI communication. We discuss the motivation of the MToM framework and its three key components that iteratively shape the human-AI communication in three stages. We then describe two empirical studies inspired by the MToM framework to demonstrate the power of MToM in guiding the design and understanding of human-AI communication. Finally, we discuss future research opportunities in human-AI interaction through the lens of MToM.