mlsanityData to 14 Oct 2026

NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models

arXiv 2305.16986 ↗First posted 26 May 2023cs.CV cs.AI cs.CLcited by 280

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Trained with an unprecedented scale of data, large language models (LLMs) like ChatGPT and GPT-4 exhibit the emergence of significant reasoning abilities from model scaling. Such a trend underscored the potential of training LLMs with unlimited language data, advancing the development of a universal embodied agent. In this work, we introduce the NavGPT, a purely LLM-based instruction-following navigation agent, to reveal the reasoning capability of GPT models in complex embodied scenes by performing zero-shot sequential action prediction for vision-and-language navigation (VLN). At each step, NavGPT takes the textual descriptions of visual observations, navigation history, and future explorable directions as inputs to reason the agent's current status, and makes the decision to approach the target. Through comprehensive experiments, we demonstrate NavGPT can explicitly perform high-level planning for navigation, including decomposing instruction into sub-goal, integrating commonsense knowledge relevant to navigation task resolution, identifying landmarks from observed scenes, tracking navigation progress, and adapting to exceptions with plan adjustment. Furthermore, we show that LLMs is capable of generating high-quality navigational instructions from observations and actions along a path, as well as drawing accurate top-down metric trajectory given the agent's navigation history. Despite the performance of using NavGPT to zero-shot R2R tasks still falling short of trained models, we suggest adapting multi-modality inputs for LLMs to use as visual navigation agents and applying the explicit reasoning of LLMs to benefit learning-based models.

Key facts. Cited by 280 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 66 of its 85 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

66 of 85 references.

References · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2015Fast R-CNNcited by 2,933
38 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  7. 2021Grounded Language-Image Pre-trainingcited by 1,156
  8. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  9. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  10. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 441
  11. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  12. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  13. 2022Visual Language Maps for Robot Navigationcited by 365
  14. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  15. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  16. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  17. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  18. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  19. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  20. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  21. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  22. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  23. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  24. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  25. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  26. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  27. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  28. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
  29. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 156
  30. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 132
  31. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
  32. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 127
  33. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  34. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 116
  35. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  36. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  37. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  38. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 18; 2024: 68; 2025: 93; 2026: 101.

280 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  3. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  4. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  5. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 156
  6. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  7. 2023Large Language Models for Robotics: A Surveycited by 133
  8. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  9. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 118
  10. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 117
  11. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 110
  12. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 109
38 more
  1. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 98
  2. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 79
  3. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 73
  4. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  5. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  6. 2024Volumetric Environment Representation for Vision-Language Navigationcited by 63
  7. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 63
  8. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 62
  9. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 62
  10. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 57
  11. 2024AffordanceLLM: Grounding Affordance from Vision Language Modelscited by 55
  12. 2025Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Modelscited by 54
  13. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 50
  14. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  15. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 46
  16. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generationcited by 44
  17. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 43
  18. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 42
  19. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 41
  20. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answeringcited by 37
  21. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  22. 2025OctoNav: Towards Generalist Embodied Navigationcited by 34
  23. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 33
  24. 2025UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learningcited by 31
  25. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  26. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  27. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 27
  28. 2024TANGO: Training-free Embodied AI Agents for Open-world Taskscited by 27
  29. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 27
  30. 2024Odyssey: Empowering Minecraft Agents with Open-World Skillscited by 26
  31. 2024An Interactive Agent Foundation Modelcited by 25
  32. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25
  33. 2024Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructionscited by 25
  34. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  35. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 25
  36. 2025PUGS: Zero-shot Physical Understanding with Gaussian Splattingcited by 24
  37. 2025Foundation Models for Spatio-Temporal Data Science: A Tutorial and Surveycited by 24
  38. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24

50 most cited of 280 All 280 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation
  2. 2025Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs
  3. 2026NavGPT-3: Harnessing Context in a Hierarchical Navigation Runtime
  4. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  5. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation
  6. 2026VISOR: VIsual Spatial Object Reasoning for Language-driven Object Navigation
  7. 2026ABot-N1: Toward a General Visual Language Navigation Foundation Model
  8. 2026FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-Language Navigation