# Latent Theory of Mind: A Decentralized Diffusion Architecture for Cooperative Manipulation

arXiv: https://arxiv.org/abs/2505.09144 · HTML: https://mlsanity.com/paper/2505.09144/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 14 May 2025.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 30 of 30)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [3D Diffuser Actor: Policy Diffusion with 3D Scene Representations](https://mlsanity.com/paper/2402.10885/) (2024)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs](https://mlsanity.com/paper/2202.04579/) (2022)
- [A Review of Cooperative Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1908.03963/) (2019)
- [3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations](https://mlsanity.com/paper/2403.03954/) (2024)
- [Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2403.03890/) (2024)
- [Diffusion Models for Reinforcement Learning: A Survey](https://mlsanity.com/paper/2311.01223/) (2023)
- [Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models](https://mlsanity.com/paper/2401.03360/) (2023)
- [PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning](https://mlsanity.com/paper/1809.03531/) (2018)
- [MADiff: Offline Multi-agent Learning with Diffusion Models](https://mlsanity.com/paper/2305.17330/) (2023)
- [EquiBot: SIM(3)-Equivariant Diffusion Policy for Generalizable and Data Efficient Learning](https://mlsanity.com/paper/2407.01479/) (2024)
- [Equivariant Diffusion Policy](https://mlsanity.com/paper/2407.01812/) (2024)
- [Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learning](https://mlsanity.com/paper/2407.01531/) (2024)
- [TacDiffusion: Force-domain Diffusion Policy for Precise Tactile Manipulation](https://mlsanity.com/paper/2409.11047/) (2024)
- [Learning a Decentralized Multi-arm Motion Planner](https://mlsanity.com/paper/2011.02608/) (2020)
- [Diffusion Trajectory-guided Policy for Long-horizon Robot Manipulation](https://mlsanity.com/paper/2502.10040/) (2025)
- [Fast, High-Quality Dual-Arm Rearrangement in Synchronous, Monotone Tabletop Setups](https://mlsanity.com/paper/1810.12202/) (2018)
- [Social Behavior as a Key to Learning-based Multi-Agent Pathfinding Dilemmas](https://mlsanity.com/paper/2408.03063/) (2024)
- [SIGMA: Sheaf-Informed Geometric Multi-Agent Pathfinding](https://mlsanity.com/paper/2502.06440/) (2025)
- [Hefty: A Modular Reconfigurable Robot for Advancing Robot Manipulation in Agriculture](https://mlsanity.com/paper/2402.18710/) (2024)
- [CCDP: Composition of Conditional Diffusion Policies with Guided Sampling](https://mlsanity.com/paper/2503.15386/) (2025)

## Built on by (most cited first, 9 of 9)

- [MIMIC-D: Multi-modal Imitation for MultI-agent Coordination with Decentralized Diffusion Policies](https://mlsanity.com/paper/2509.14159/) (2025)
- [CHORUS: Decentralized Multi-Embodiment Collaboration with One VLA Policy](https://mlsanity.com/paper/2606.12352/) (2026)
- [FALCON: Actively Decoupled Visuomotor Policies for Loco-Manipulation with Foundation-Model-Based Coordination](https://mlsanity.com/paper/2512.04381/) (2025)
- [PATCH: Action-Chunk-Conditioned Latent Patch Innovation Monitoring for Robot Manipulation](https://mlsanity.com/paper/2606.16690/) (2026)
- [A formal definition and meta-model for a machine theory of mind](https://mlsanity.com/paper/2606.03471/) (2026)
- [CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration](https://mlsanity.com/paper/2609.34782/) (2026)
- [DuoMind: Enabling Distributed Multi-Robot Coordination with Semantic Communication](https://mlsanity.com/paper/2610.02161/) (2026)
- [Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination](https://mlsanity.com/paper/2610.02170/) (2026)
- [REACT: Rolling Denoising and Dual Decoupling for Reactive Robot Control with VLA Models](https://mlsanity.com/paper/2610.12007/) (2026)

## Abstract (arXiv)

We present Latent Theory of Mind (LatentToM), a decentralized diffusion policy architecture for collaborative robot manipulation. Our policy allows multiple manipulators with their own perception and computation to collaborate with each other towards a common task goal with or without explicit communication. Our key innovation lies in allowing each agent to maintain two latent representations: an ego embedding specific to the robot, and a consensus embedding trained to be common to both robots, despite their different sensor streams and poses. We further let each robot train a decoder to infer the other robot's ego embedding from their consensus embedding, akin to theory of mind in latent space. Training occurs centrally, with all the policies' consensus encoders supervised by a loss inspired by sheaf theory, a mathematical theory for clustering data on a topological manifold. Specifically, we introduce a first-order cohomology loss to enforce sheaf-consistent alignment of the consensus embeddings. To preserve the expressiveness of the consensus embedding, we further propose structural constraints based on theory of mind and a directional consensus mechanism. Execution can be fully distributed, requiring no explicit communication between policies. In which case, the information is exchanged implicitly through each robot's sensor stream by observing the actions of the other robots and their effects on the scene. Alternatively, execution can leverage direct communication to share the robots' consensus embeddings, where the embeddings are shared once during each inference step and are aligned using the sheaf Laplacian. In our hardware experiments, LatentToM outperforms a naive decentralized diffusion baseline, and shows comparable performance with a state-of-the-art centralized diffusion policy for bi-manual manipulation. Project website: https://stanfordmsl.github.io/LatentToM/.