# Two Body Problem: Collaborative Visual Task Completion

arXiv: https://arxiv.org/abs/1904.05879 · HTML: https://mlsanity.com/paper/1904.05879/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 30 of 45)

- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [Simple but Effective: CLIP Embeddings for Embodied AI](https://mlsanity.com/paper/2111.09888/) (2021)
- [Visual Room Rearrangement](https://mlsanity.com/paper/2103.16544/) (2021)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNav](https://mlsanity.com/paper/2301.07302/) (2023)
- [MindAgent: Emergent Gaming Interaction](https://mlsanity.com/paper/2309.09971/) (2023)
- [ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes](https://mlsanity.com/paper/2304.04321/) (2023)
- [LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Models](https://mlsanity.com/paper/2310.03903/) (2023)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistant](https://mlsanity.com/paper/2203.04203/) (2022)
- [Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models](https://mlsanity.com/paper/2310.19619/) (2023)
- [LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environments](https://mlsanity.com/paper/2407.10031/) (2024)
- [Pretrained Language Models as Visual Planners for Human Assistance](https://mlsanity.com/paper/2304.09179/) (2023)
- [The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation](https://mlsanity.com/paper/2108.11550/) (2021)
- [Last-Mile Embodied Visual Navigation](https://mlsanity.com/paper/2211.11746/) (2022)
- [GridToPix: Training Embodied Agents with Minimal Supervision](https://mlsanity.com/paper/2105.00931/) (2021)
- [HandMeThat: Human-Robot Communication in Physical and Social Environments](https://mlsanity.com/paper/2310.03779/) (2023)
- [Learning Efficient Multi-Agent Cooperative Visual Exploration](https://mlsanity.com/paper/2110.05734/) (2021)
- [Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents](https://mlsanity.com/paper/2110.05769/) (2021)
- [Collaborative Visual Navigation](https://mlsanity.com/paper/2107.01151/) (2021)
- [Semantic Tracklets: An Object-Centric Representation for Visual Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2108.03319/) (2021)
- [Shaping embodied agent behavior with activity-context priors from egocentric video](https://mlsanity.com/paper/2110.07692/) (2021)
- [Adaptive Coordination in Social Embodied Rearrangement](https://mlsanity.com/paper/2306.00087/) (2023)
- [Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environments](https://mlsanity.com/paper/2509.22698/) (2025)
- [MOPA: Modular Object Navigation with PointGoal Agents](https://mlsanity.com/paper/2304.03696/) (2023)

## Abstract (arXiv)

Collaboration is a necessary skill to perform tasks that are beyond one agent's capabilities. Addressed extensively in both conventional and modern AI, multi-agent collaboration has often been studied in the context of simple grid worlds. We argue that there are inherently visual aspects to collaboration which should be studied in visually rich environments. A key element in collaboration is communication that can be either explicit, through messages, or implicit, through perception of the other agents and the visual world. Learning to collaborate in a visual environment entails learning (1) to perform the task, (2) when and what to communicate, and (3) how to act based on these communications and the perception of the visual world. In this paper we study the problem of learning to collaborate directly from pixels in AI2-THOR and demonstrate the benefits of explicit and implicit modes of communication to perform visual tasks. Refer to our project page for more details: https://prior.allenai.org/projects/two-body-problem