mlsanityData to 14 Oct 2026

NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models

arXiv 2407.12366 ↗First posted 17 Jul 2024cs.CV cs.AI cs.CLcited by 98

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Capitalizing on the remarkable advancements in Large Language Models (LLMs), there is a burgeoning initiative to harness LLMs for instruction following robotic navigation. Such a trend underscores the potential of LLMs to generalize navigational reasoning and diverse language understanding. However, a significant discrepancy in agent performance is observed when integrating LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous downstream specialist models. Furthermore, the inherent capacity of language to interpret and facilitate communication in agent interactions is often underutilized in these integrations. In this work, we strive to bridge the divide between VLN-specialized models and LLM-based navigation paradigms, while maintaining the interpretative prowess of LLMs in generating linguistic navigational reasoning. By aligning visual content in a frozen LLM, we encompass visual observation comprehension for LLMs and exploit a way to incorporate LLMs and navigation policy networks for effective action predictions and navigational reasoning. We demonstrate the data efficiency of the proposed methods and eliminate the gap between LM-based agents and state-of-the-art VLN specialists.

Key facts. Cited by 98 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 68 of its 78 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

68 of 78 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains17 May
  2. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning12 Mar
  3. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models26 May

Other references · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
38 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  3. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  7. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  9. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  11. 2018On Evaluation of Embodied Navigation Agentscited by 547
  12. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  13. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  14. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
  15. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  16. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 333
  17. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  18. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  19. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
  20. 2019Vision-and-Dialog Navigationcited by 250
  21. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  22. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  23. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
  24. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
  25. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 179
  26. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  27. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 176
  28. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 157
  29. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  30. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 130
  31. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 124
  32. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 117
  33. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 116
  34. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 110
  35. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 105
  36. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 105
  37. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  38. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 48; 2026: 42.

98 papers.

Sort:

By area:

  1. 2024NaVILA: Legged Robot Vision-Language-Action Model for Navigationcited by 211
  2. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 67
  3. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 63
  4. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 50
  5. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 47
  6. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 46
  7. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 40
  8. 2025OctoNav: Towards Generalist Embodied Navigationcited by 34
  9. 2025UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learningcited by 31
  10. 2024TANGO: Training-free Embodied AI Agents for Open-world Taskscited by 27
  11. 2025Latent Action Pretraining Through World Modelingcited by 24
  12. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
38 more
  1. 2024RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-Worldcited by 22
  2. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 20
  3. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 19
  4. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 17
  5. 2025Towards Physically Executable 3D Gaussian for Embodied Navigationcited by 17
  6. 2025Thinking in 360deg: Humanoid Visual Search in the Wildcited by 15
  7. 2025Do Visual Imaginations Improve Vision-and-Language Navigation Agents?cited by 14
  8. 2025SkyVLN: Vision-and-Language Navigation and NMPC Control for UAVs in Urban Environmentscited by 14
  9. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  10. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigationcited by 13
  11. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  12. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 12
  13. 2025SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesiscited by 12
  14. 2026Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigationcited by 11
  15. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 11
  16. 2025GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigationcited by 10
  17. 2026VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memorycited by 10
  18. 2025Meta-Memory: Retrieving and Integrating Semantic-Spatial Memories for Robot Spatial Reasoningcited by 9
  19. 2026SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigationcited by 9
  20. 2025CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?cited by 8
  21. 2025VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Modelscited by 7
  22. 2025Active Test-time Vision-Language Navigationcited by 7
  23. 2025TANGO: Traversability-Aware Navigation with Local Metric Control for Topological Goalscited by 7
  24. 2025NavQ: Learning a Q-Model for Foresighted Vision-and-Language Navigationcited by 6
  25. 2026ABot-N1: Toward a General Visual Language Navigation Foundation Modelcited by 6
  26. 2024Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrievalcited by 5
  27. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  28. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agentscited by 5
  29. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  30. 2025SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuningcited by 5
  31. 2026APEX: A Decoupled Memory-based Explorer for Asynchronous Aerial Object Goal Navigationcited by 5
  32. 2026One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representationcited by 5
  33. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  34. 2025Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigationcited by 4
  35. 2025DAgger Diffusion Navigation: DAgger Boosted Diffusion Policy for Vision-Language Navigationcited by 4
  36. 2025MSNav: Zero-Shot Vision-and-Language Navigation with Dynamic Memory and LLM Spatial Reasoningcited by 4
  37. 2026FineCog-Nav: Integrating Fine-grained Cognitive Modules for Zero-shot Multimodal UAV Navigationcited by 4
  38. 2026ESARBench: A Benchmark for Agentic UAV Embodied Search and Rescuecited by 4

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs
  2. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  3. 2025Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation
  4. 2025UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Model
  5. 2026DV-VLN: Dual Verification for Reliable LLM-Based Vision-and-Language Navigation
  6. 2025Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models
  7. 2026Learning to Retrieve Navigable Candidates for Efficient Vision-and-Language Navigation
  8. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigation