mlsanityData to 7 Oct 2026

Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents

arXiv 2110.05769 ↗First posted 12 Oct 2021cs.CV cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Communication between embodied AI agents has received increasing attention in recent years. Despite its use, it is still unclear whether the learned communication is interpretable and grounded in perception. To study the grounding of emergent forms of communication, we first introduce the collaborative multi-object navigation task CoMON. In this task, an oracle agent has detailed environment information in the form of a map. It communicates with a navigator agent that perceives the environment visually and is tasked to find a sequence of goals. To succeed at the task, effective communication is essential. CoMON hence serves as a basis to study different communication mechanisms between heterogeneous agents, that is, agents with different capabilities and roles. We study two common communication mechanisms and analyze their communication patterns through an egocentric and spatial lens. We show that the emergent communication can be grounded to the agent observations and the spatial structure of the 3D environment. Video summary: https://youtu.be/kLv2rxO9t0g

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 55 of its 73 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

55 of 73 references.

References · 55

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  9. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  10. 2018On Evaluation of Embodied Navigation Agentscited by 526
  11. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  12. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
38 more
  1. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  2. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  3. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  4. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  5. 2016Learning Multiagent Communication with Backpropagationcited by 355
  6. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  7. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 257
  8. 2020Experience Grounds Languagecited by 241
  9. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  10. 2017Learning with Opponent-Learning Awarenesscited by 218
  11. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  12. 2020Neural Topological SLAM for Visual Navigationcited by 193
  13. 2016Multi-Agent Cooperation and the Emergence of (Natural) Languagecited by 181
  14. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  15. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  16. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 168
  17. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  18. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 143
  19. 2020Who2com: Collaborative Perception via Learnable Handshake Communicationcited by 127
  20. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  21. 2021Visual Room Rearrangementcited by 101
  22. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  23. 2017Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialogcited by 86
  24. 2019Multi-Target Embodied Question Answeringcited by 74
  25. 2017Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learningcited by 70
  26. 2019On the Pitfalls of Measuring Emergent Communicationcited by 66
  27. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60
  28. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  29. 2020AllenAct: A Framework for Embodied AI Researchcited by 54
  30. 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
  31. 2017A Dataset for Developing and Benchmarking Active Visioncited by 50
  32. 2019PIC: Permutation Invariant Critic for Multi-Agent Deep Reinforcement Learningcited by 46
  33. 2019Two Body Problem: Collaborative Visual Task Completioncited by 45
  34. 2019Coordinated Exploration via Intrinsic Rewards for Multi-Agent Reinforcement Learningcited by 30
  35. 2020EgoMap: Projective mapping and structured egocentric memory for Deep RLcited by 23
  36. 2016Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networkscited by 21
  37. 2017HoME: a Household Multimodal Environmentcited by 20
  38. 2021GridToPix: Training Embodied Agents with Minimal Supervisioncited by 20

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 9; 2024: 2; 2025: 1; 2026: 3.

19 papers.

Sort:

By area:

  1. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  2. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
  3. 2022Retrospectives on the Embodied AI Workshopcited by 46
  4. 2022Last-Mile Embodied Visual Navigationcited by 25
  5. 2022Learning Active Camera for Multi-Object Navigationcited by 20
  6. 2024A Review of the Applications of Deep Learning-Based Emergent Communicationcited by 17
  7. 2023Adaptive Coordination in Social Embodied Rearrangementcited by 14
  8. 2023EC^2: Emergent Communication for Embodied Controlcited by 10
  9. 2023Exploitation-Guided Exploration for Semantic Embodied Navigationcited by 9
  10. 2022What do navigation agents learn about their environment?cited by 8
  11. 2023Chat2Map: Efficient Scene Mapping from Multi-Ego Conversationscited by 8
  12. 2023Natural Language-conditioned Reinforcement Learning with Inside-out Task Language Development and Translationcited by 7
7 more
  1. 2023Multi-Level Compositional Reasoning for Interactive Instruction Followingcited by 7
  2. 2023Heterogeneous Embodied Multi-Agent Collaborationcited by 6
  3. 2025From Grunts to Lexicons: Emergent Language from Cooperative Foragingcited by 1
  4. 2026Learning to Communicate Across Modalities: Perceptual Heterogeneity in Multi-Agent Systemscited by 1
  5. 2026Unified Visuomotor Targets: Supervising VLAs Beyond Physical Actionscited by 1
  6. 2024Improving Zero-Shot ObjectNav with Generative Communicationcited by 0
  7. 2026Emergent Language as an Approach to Conscious AIcited by 0