# DPMT: Dual Process Multi-scale Theory of Mind Framework for Real-time Human-AI Collaboration

arXiv: https://arxiv.org/abs/2507.14088 · HTML: https://mlsanity.com/paper/2507.14088/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 18 Jul 2025.

## Builds on (most cited first, 17 of 17)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models](https://mlsanity.com/paper/2401.13919/) (2024)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [ProAgent: Building Proactive Cooperative Agents with Large Language Models](https://mlsanity.com/paper/2308.11339/) (2023)
- [ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind](https://mlsanity.com/paper/2111.09189/) (2021)
- [LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination](https://mlsanity.com/paper/2312.15224/) (2023)
- [On the Critical Role of Conventions in Adaptive Human-AI Collaboration](https://mlsanity.com/paper/2104.02871/) (2021)
- [Mutual Theory of Mind in Human-AI Collaboration: An Empirical Study with LLM-driven AI Agents in a Real-time Shared Workspace Task](https://mlsanity.com/paper/2409.08811/) (2024)
- [Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks](https://mlsanity.com/paper/2007.12803/) (2020)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [Mixture of personality improved Spiking actor network for efficient multi-agent cooperation](https://mlsanity.com/paper/2305.05898/) (2023)

## Abstract (arXiv)

Real-time human-artificial intelligence (AI) collaboration is crucial yet challenging, especially when AI agents must adapt to diverse and unseen human behaviors in dynamic scenarios. Existing large language model (LLM) agents often fail to accurately model the complex human mental characteristics such as domain intentions, especially in the absence of direct communication. To address this limitation, we propose a novel dual process multi-scale theory of mind (DPMT) framework, drawing inspiration from cognitive science dual process theory. Our DPMT framework incorporates a multi-scale theory of mind (ToM) module to facilitate robust human partner modeling through mental characteristic reasoning. Experimental results demonstrate that DPMT significantly enhances human-AI collaboration, and ablation studies further validate the contributions of our multi-scale ToM in the slow system.