# CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views

arXiv: https://arxiv.org/abs/2607.06691 · HTML: https://mlsanity.com/paper/2607.06691/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 97 references that are in the map.
- First posted on arXiv 7 Jul 2026.

## Builds on (most cited first, 19 of 19)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos](https://mlsanity.com/paper/2411.19167/) (2024)
- [PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks](https://mlsanity.com/paper/2411.00081/) (2024)
- [RED: Reinforced Encoder-Decoder Networks for Action Anticipation](https://mlsanity.com/paper/1707.04818/) (2017)
- [SAM 3D Body: Robust Full-Body Human Mesh Recovery](https://mlsanity.com/paper/2602.15989/) (2026)
- [EgoHumans: An Egocentric 3D Multi-Human Benchmark](https://mlsanity.com/paper/2305.16487/) (2023)
- [CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object REarrangement](https://mlsanity.com/paper/2406.19353/) (2024)
- [Embody 3D: A Large-scale Multimodal Motion and Behavior Dataset](https://mlsanity.com/paper/2510.16258/) (2025)
- [Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023](https://mlsanity.com/paper/2306.16545/) (2023)
- [MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems](https://mlsanity.com/paper/2505.18943/) (2025)
- [MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mind](https://mlsanity.com/paper/2507.04415/) (2025)
- [EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos](https://mlsanity.com/paper/2503.22152/) (2025)
- [EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World](https://mlsanity.com/paper/2501.19061/) (2025)
- [EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models](https://mlsanity.com/paper/2506.01608/) (2025)
- [HiMemFormer: Hierarchical Memory-Aware Transformer for Multi-Agent Action Anticipation](https://mlsanity.com/paper/2411.01455/) (2024)

## Abstract (arXiv)

Human-human collaboration is a fundamental aspect of everyday life, essential to success in a wide range of goal-directed activities from household tasks to professional teamwork. While much research has focused on modeling coordination and task execution, the cognitive processes that support such collaboration, particularly Theory of Mind (the ability to infer the mental states of others), remain difficult to study in natural settings. To address this gap, we introduce a novel egocentric and exocentric video dataset capturing real-world collaboration in cooking scenarios. The dataset integrates multi-perspective video, high-quality audio, gaze tracking, and 3D scene and object scans, with annotations for shared attention to objects, social cues and interactions between agents, as well as agent-object interactions. We establish benchmarks for Joint Attention Estimation, Socially Conditioned Object Interaction Anticipation, and Collaborative Handover Prediction, enabling research on multimodal perception, proactive assistance, and collaborative planning. By providing temporally aligned, richly annotated multimodal data, CoMind facilitates the development and evaluation of AI systems capable of modeling complex social interactions and reasoning about human behaviors in collaborative environments. Our dataset and benchmarks are made available at https://comind.ethz.ch/.