mlsanityData to 7 Oct 2026

MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigation

arXiv 2401.07314 ↗First posted 14 Jan 2024cs.AI cs.CV cs.ROcited by 85

Not on the map. Why?

Abstract (arXiv)

Embodied agents equipped with GPT as their brains have exhibited extraordinary decision-making and generalization abilities across various tasks. However, existing zero-shot agents for vision-and-language navigation (VLN) only prompt GPT-4 to select potential locations within localized environments, without constructing an effective "global-view" for the agent to understand the overall environment. In this work, we present a novel map-guided GPT-based agent, dubbed MapGPT, which introduces an online linguistic-formed map to encourage global exploration. Specifically, we build an online map and incorporate it into the prompts that include node information and topological relationships, to help GPT understand the spatial environment. Benefiting from this design, we further propose an adaptive planning mechanism to assist the agent in performing multi-step path planning based on a map, systematically exploring multiple candidate nodes or sub-goals step by step. Extensive experiments demonstrate that our MapGPT is applicable to both GPT-4 and GPT-4V, achieving state-of-the-art zero-shot performance on R2R and REVERIE simultaneously (~10% and ~12% improvements in SR), and showcasing the newly emergent global thinking and path planning abilities of the GPT.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 14 Jan 2024.

Builds on

35 of 52 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  12. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
23 more
  1. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  2. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  3. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  4. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  5. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  6. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  7. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  8. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  9. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  10. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  11. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  12. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  13. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  14. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  15. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  16. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  17. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  18. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  19. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  20. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  21. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  22. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  23. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 30; 2026: 44.

85 papers.

Sort:

By area:

  1. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 201
  2. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  3. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  4. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 43
  5. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 33
  6. 2025OctoNav: Towards Generalist Embodied Navigationcited by 33
  7. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  8. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  9. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  10. 2025DreamNav: A Trajectory-Based Imaginative Framework for Zero-Shot Vision-and-Language Navigationcited by 15
  11. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  12. 2024Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Studycited by 12
38 more
  1. 2024Guide-LLM: An Embodied LLM Agent and Text-Based Topological Map for Robotic Guidance of People with Visual Impairmentscited by 10
  2. 2025GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigationcited by 10
  3. 2024Multimodal LLM Guided Exploration and Active Mapping using Fisher Informationcited by 9
  4. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  5. 2026SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigationcited by 8
  6. 2025Active Test-time Vision-Language Navigationcited by 7
  7. 2025Aerial Vision-Language Navigation with a Unified Framework for Spatial, Temporal and Embodied Reasoningcited by 7
  8. 2024Evaluating Vision-Language Models as Evaluators in Path Planningcited by 6
  9. 2025PanoGen++: Domain-Adapted Text-Guided Panoramic Environment Generation for Vision-and-Language Navigationcited by 6
  10. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  11. 2025Fast-SmartWay: Panoramic-Free End-to-End Zero-Shot Vision-and-Language Navigationcited by 6
  12. 2026Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigationcited by 6
  13. 2026AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harnesscited by 6
  14. 2024Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrievalcited by 5
  15. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  16. 2026One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representationcited by 5
  17. 2024Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigationcited by 4
  18. 2025Endowing Embodied Agents with Spatial Reasoning Capabilities for Vision-and-Language Navigationcited by 4
  19. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  20. 2025Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigationcited by 4
  21. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agentscited by 4
  22. 2025DAgger Diffusion Navigation: DAgger Boosted Diffusion Policy for Vision-Language Navigationcited by 4
  23. 2025Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Promptingcited by 4
  24. 2026Trajectory-Diversity-Driven Robust Vision-and-Language Navigationcited by 4
  25. 2026Three-Step Nav: A Hierarchical Global-Local Planner for Zero-Shot Vision-and-Language Navigationcited by 4
  26. 2025Mem4Nav: Boosting Vision-and-Language Navigation in Urban Environments with a Hierarchical Spatial-Cognition Long-Short Memory Systemcited by 3
  27. 2025Vision-and-Language Navigation with Analogical Textual Descriptions in LLMscited by 3
  28. 2026Structured Observation Language for Efficient and Generalizable Vision-Language Navigationcited by 3
  29. 2026Stop Wandering: Efficient Vision-Language Navigation via Metacognitive Reasoningcited by 3
  30. 2026LookasideVLN: Direction-Aware Aerial Vision-and-Language Navigationcited by 3
  31. 2026LLM-Guided Agentic Floor Plan Parsing for Accessible Indoor Navigation of Blind and Low-Vision Peoplecited by 3
  32. 2026Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigationcited by 3
  33. 2026SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoningcited by 3
  34. 2026CoFL: Continuous Flow Fields for Language-Conditioned Navigationcited by 2
  35. 2026TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigationcited by 2
  36. 2026Does Peer Observation Help? Vision-Sharing Collaboration for Vision-Language Navigationcited by 2
  37. 2026P2DNav: Panorama-to-Downview Reasoning for Zero-shot Vision-and-Language Navigationcited by 2
  38. 2026CLOSER-VLN: Closed-Loop Self-Verified Retrieval-Augmented Reasoning for Aerial Vision-Language Navigationcited by 2

50 most cited of 85 All 85 (JSON)