# Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation Using Vision Language Models

arXiv: https://arxiv.org/abs/2310.07937 · HTML: https://mlsanity.com/paper/2310.07937/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 35 of its 60 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 35)

- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [Object Goal Navigation using Goal-Oriented Semantic Exploration](https://mlsanity.com/paper/2007.00643/) (2020)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [Neural Topological SLAM for Visual Navigation](https://mlsanity.com/paper/2005.12256/) (2020)
- [Simple but Effective: CLIP Embeddings for Embodied AI](https://mlsanity.com/paper/2111.09888/) (2021)
- [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021)
- [ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings](https://mlsanity.com/paper/2206.12403/) (2022)
- [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2307.04738/) (2023)
- [PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning](https://mlsanity.com/paper/2201.10029/) (2022)
- [L3MVN: Leveraging Large Language Models for Visual Target Navigation](https://mlsanity.com/paper/2304.05501/) (2023)
- [Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scale](https://mlsanity.com/paper/2204.03514/) (2022)
- [Auxiliary Tasks and Exploration Enable ObjectNav](https://mlsanity.com/paper/2104.04112/) (2021)
- [Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation](https://mlsanity.com/paper/2202.02440/) (2022)
- [SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation](https://mlsanity.com/paper/2012.04512/) (2020)

## Built on by (most cited first, 8 of 8)

- [Enhancing Multi-Robot Semantic Navigation Through Multimodal Chain-of-Thought Score Collaboration](https://mlsanity.com/paper/2412.18292/) (2024)
- [Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Survey](https://mlsanity.com/paper/2504.14520/) (2025)
- [MAPLE: Multi-Agent Adaptive Planning with Long-Term Memory for Table Reasoning](https://mlsanity.com/paper/2506.05813/) (2025)
- [Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2605.18431/) (2026)
- [A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications](https://mlsanity.com/paper/2608.20379/) (2026)
- [Embodied Spatial Intelligence: from Implicit Scene Modeling to Spatial Reasoning](https://mlsanity.com/paper/2509.00465/) (2025)
- [Semantic Area Graph Reasoning for Multi-Robot Language-Guided Search](https://mlsanity.com/paper/2604.16263/) (2026)
- [PEACE: A Planner-Executor Agent with Constraint Enforcement for UAVs](https://mlsanity.com/paper/2606.00104/) (2026)

## Abstract (arXiv)

Visual target navigation is a critical capability for autonomous robots operating in unknown environments, particularly in human-robot interaction scenarios. While classical and learning-based methods have shown promise, most existing approaches lack common-sense reasoning and are typically designed for single-robot settings, leading to reduced efficiency and robustness in complex environments. To address these limitations, we introduce Co-NavGPT, a novel framework that integrates a Vision Language Model (VLM) as a global planner to enable common-sense multi-robot visual target navigation. Co-NavGPT aggregates sub-maps from multiple robots with diverse viewpoints into a unified global map, encoding robot states and frontier regions. The VLM uses this information to assign frontiers across the robots, facilitating coordinated and efficient exploration. Experiments on the Habitat-Matterport 3D (HM3D) demonstrate that Co-NavGPT outperforms existing baselines in terms of success rate and navigation efficiency, without requiring task-specific training. Ablation studies further confirm the importance of semantic priors from the VLM. We also validate the framework in real-world scenarios using quadrupedal robots. Supplementary video and code are available at: https://sites.google.com/view/co-navgpt2.