# Learning Efficient Multi-Agent Cooperative Visual Exploration

arXiv: https://arxiv.org/abs/2110.05734 · HTML: https://mlsanity.com/paper/2110.05734/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 54 of its 92 references that are in the map.
- First posted on arXiv 12 Oct 2021.

## Builds on (most cited first, 30 of 54)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Neural Topological SLAM for Visual Navigation](https://mlsanity.com/paper/2005.12256/) (2020)
- [Learning Attentional Communication for Multi-Agent Cooperation](https://mlsanity.com/paper/1805.07733/) (2018)
- [Graph Convolutional Reinforcement Learning](https://mlsanity.com/paper/1810.09202/) (2018)
- [Multi-task Deep Reinforcement Learning with PopArt](https://mlsanity.com/paper/1809.04474/) (2018)
- [Visual Semantic Navigation using Scene Priors](https://mlsanity.com/paper/1810.06543/) (2018)
- [Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications](https://mlsanity.com/paper/1812.11794/) (2018)
- [Learning Exploration Policies for Navigation](https://mlsanity.com/paper/1903.01959/) (2019)
- [Episodic Curiosity through Reachability](https://mlsanity.com/paper/1810.02274/) (2018)
- [Scene Memory Transformer for Embodied Agents in Long-Horizon Tasks](https://mlsanity.com/paper/1903.03878/) (2019)

## Built on by (most cited first, 19 of 19)

- [GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoning](https://mlsanity.com/paper/2505.23399/) (2025)
- [Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration](https://mlsanity.com/paper/2301.03398/) (2023)
- [Transformers in Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.05979/) (2023)
- [Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering](https://mlsanity.com/paper/2311.17331/) (2023)
- [Multi-Robot Active Mapping via Neural Bipartite Graph Matching](https://mlsanity.com/paper/2203.16319/) (2022)
- [DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization](https://mlsanity.com/paper/2207.05631/) (2022)
- [Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation Using Vision Language Models](https://mlsanity.com/paper/2310.07937/) (2023)
- [Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigation](https://mlsanity.com/paper/2302.04094/) (2023)
- [Heterogeneous Embodied Multi-Agent Collaboration](https://mlsanity.com/paper/2307.13957/) (2023)
- [Visual Hindsight Self-Imitation Learning for Interactive Navigation](https://mlsanity.com/paper/2312.03446/) (2023)
- [Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challenges](https://mlsanity.com/paper/2305.10091/) (2023)
- [SayCoNav: Utilizing Large Language Models for Adaptive Collaboration in Decentralized Multi-Robot Navigation](https://mlsanity.com/paper/2505.13729/) (2025)
- [Boosting Team Modeling through Tempo-Relational Representation Learning](https://mlsanity.com/paper/2507.13305/) (2025)
- [Distributed NeRF Learning for Collaborative Multi-Robot Perception](https://mlsanity.com/paper/2409.20289/) (2024)
- [Symmetry-aware Neural Architecture for Embodied Visual Navigation](https://mlsanity.com/paper/2112.09515/) (2021)
- [Active Neural Topological Mapping for Multi-Agent Exploration](https://mlsanity.com/paper/2311.00252/) (2023)
- [Diverse Policies Converge in Reward-free Markov Decision Processe](https://mlsanity.com/paper/2308.11924/) (2023)
- [MASP: Scalable GNN-based Planning for Multi-Agent Navigation](https://mlsanity.com/paper/2312.02522/) (2023)
- [Systematic Multi-Agent Vision-and-Language Navigation: Formulation, Benchmark, and Method](https://mlsanity.com/paper/2609.35965/) (2026)

## Abstract (arXiv)

We tackle the problem of cooperative visual exploration where multiple agents need to jointly explore unseen regions as fast as possible based on visual signals. Classical planning-based methods often suffer from expensive computation overhead at each step and a limited expressiveness of complex cooperation strategy. By contrast, reinforcement learning (RL) has recently become a popular paradigm for tackling this challenge due to its modeling capability of arbitrarily complex strategies and minimal inference overhead. In this paper, we extend the state-of-the-art single-agent visual navigation method, Active Neural SLAM (ANS), to the multi-agent setting by introducing a novel RL-based planning module, Multi-agent Spatial Planner (MSP).MSP leverages a transformer-based architecture, Spatial-TeamFormer, which effectively captures spatial relations and intra-agent interactions via hierarchical spatial self-attentions. In addition, we also implement a few multi-agent enhancements to process local information from each agent for an aligned spatial representation and more precise planning. Finally, we perform policy distillation to extract a meta policy to significantly improve the generalization capability of final policy. We call this overall solution, Multi-Agent Active Neural SLAM (MAANS). MAANS substantially outperforms classical planning-based baselines for the first time in a photo-realistic 3D simulator, Habitat. Code and videos can be found at https://sites.google.com/view/maans.