mlsanityData to 7 Oct 2026

Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Task

arXiv 2409.08811 ↗First posted 13 Sep 2024cs.HC cs.AI cs.MAcited by 17

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Theory of Mind (ToM) significantly impacts human collaboration and communication as a crucial capability to understand others. When AI agents with ToM capability collaborate with humans, Mutual Theory of Mind (MToM) arises in such human-AI teams (HATs). The MToM process, which involves interactive communication and ToM-based strategy adjustment, affects the team's performance and collaboration process. To explore the MToM process, we conducted a mixed-design experiment using a large language model-driven AI agent with ToM and communication modules in a real-time shared-workspace task. We find that the agent's ToM capability does not significantly impact team performance but enhances human understanding of the agent and the feeling of being understood. Most participants in our study believe verbal communication increases human burden, and the results show that bidirectional communication leads to lower HAT performance. We discuss the results' implications for designing AI agents that collaborate with humans in real-time shared workspace tasks.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 41 of its 130 references that are in the map. First posted on arXiv 13 Sep 2024.

Builds on

41 of 130 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination23 Dec

Other references · 40

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  8. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  9. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  10. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  11. 2018Machine Theory of Mindcited by 222
  12. 2021Collaborating with Humans without Human Datacited by 139
28 more
  1. 2017Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problemscited by 126
  2. 2024Simulating Classroom Education with LLM-Empowered Agentscited by 125
  3. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimizationcited by 93
  4. 2019Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learningcited by 68
  5. 2023LASER: LLM Agent with State-Space Exploration for Web Navigationcited by 66
  6. 2021MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Taskscited by 62
  7. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  8. 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
  9. 2023Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosiscited by 30
  10. 2021Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rolloutscited by 24
  11. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  12. 2023ZSC-Eval: An Evaluation Toolkit and Benchmark for Multi-agent Zero-shot Coordinationcited by 23
  13. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  14. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  15. 2024LLM-Personalize: Aligning LLM Planners with Human Preferences via Reinforced Self-Training for Housekeeping Robotscited by 18
  16. 2022Mutual Theory of Mind for Human-AI Communicationcited by 15
  17. 2023Math Agents: Computational Infrastructure, Mathematical Embedding, and Genomicscited by 15
  18. 2024The Effects of Embodiment and Personality Expression on Learning in LLM-based Educational Agentscited by 13
  19. 2023Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordinationcited by 12
  20. 2024LLM-enhanced Scene Graph Learning for Household Rearrangementcited by 12
  21. 2022ELIGN: Expectation Alignment as a Multi-Agent Intrinsic Rewardcited by 11
  22. 2024Why Would You Suggest That? Human Trust in Language Model Responsescited by 9
  23. 2023Fast Teammate Adaptation in the Presence of Sudden Policy Changecited by 8
  24. 2023Efficient Human-AI Coordination via Preparatory Language-based Conventioncited by 7
  25. 2024Human and LLM-Based Voice Assistant Interaction: An Analytical Framework for User Verbal and Nonverbal Behaviorscited by 5
  26. 2024An Extremely Data-efficient and Generative LLM-based Reinforcement Learning Agent for Recommenderscited by 2
  27. 2024Seeking Soulmate via Voice: Understanding Promises and Challenges of Online Synchronized Voice-Based Mobile Datingcited by 1
  28. 2024Expedient Assistance and Consequential Misunderstanding: Envisioning an Operationalized Mutual Theory of Mindcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024Position: Theory of Mind Benchmarks are Broken for Large Language Modelscited by 30
  2. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 28
  3. 2025Theory of Mind in Large Language Models: Assessment and Enhancementcited by 24
  4. 2025Leveraging Dual Process Theory in Language Agent Framework for Real-time Simultaneous Human-AI Collaborationcited by 10
  5. 2025HARBOR: Exploring Persona Dynamics in Multi-Agent Competitioncited by 6
  6. 2025AI Agent Behavioral Sciencecited by 5
  7. 2025Gricean Norms as a Basis for Effective Collaborationcited by 3
  8. 2025Who is Helping Whom? Analyzing Inter-dependencies to Evaluate Cooperation in Human-AI Teamingcited by 2
  9. 2025Mind the Motions: Benchmarking Theory-of-Mind in Everyday Body Languagecited by 1
  10. 2026Interaction-Centered Intelligence: Toward an Interaction-Based Theory of Human-AI Co-Creationcited by 1
  11. 2026What Types of Human-AI Teams Exist?cited by 1
  12. 2025Theory of Mind in Action: The Instruction Inference Task in Dynamic Human-Agent Collaborationcited by 0
5 more
  1. 2025DPMT: Dual Process Multi-scale Theory of Mind Framework for Real-time Human-AI Collaborationcited by 0
  2. 2025When Researchers Say Mental Model/Theory of Mind of AI, What Are They Really Talking About?cited by 0
  3. 2025SocialNLI: A Dialogue-Centric Social Inference Datasetcited by 0
  4. 2025ReCollab: Retrieval-Augmented LLMs for Cooperative Ad-hoc Teammate Modelingcited by 0
  5. 2026Ream: Unfolding Mutual Awareness in Human-Agent Workspacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Theory of Mind for LLM-based Multi-Agent Coordination
  2. 2025Towards Cognitive Synergy in LLM-Based Multi-Agent Systems: Integrating Theory of Mind and Critical Evaluation
  3. 2025Infusing Theory of Mind into Socially Intelligent LLM Agents
  4. 2025Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration
  5. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Models
  6. 2025Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective
  7. 2025LLM-Hanabi: Evaluating Multi-Agent Gameplays with Theory-of-Mind and Rationale Inference in Imperfect Information Collaboration Game
  8. 2026Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration