mlsanityData to 7 Oct 2026

NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning

arXiv 2403.07376 ↗First posted 12 Mar 2024cs.CV cs.AI cs.CLcited by 77

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions. Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability. However, their predominant use in an offline manner usually suffers from substantial domain gap between the VLN task and the LLM training corpus. This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision, leading to a significant mitigation of the domain gap in a cost-effective manner. Specifically, at each timestep, the LLM is prompted to forecast the navigational chain-of-thought by: 1) acting as a world model to imagine the next observation according to the instruction, 2) selecting the candidate observation that best aligns with the imagination, and 3) determining the action based on the reasoning from the prior steps. Through constructing formalized labels for training, the LLM can learn to generate desired and reasonable chain-of-thought outputs for improving the action decision. Experimental results across various training settings and popular VLN benchmarks (e.g., Room-to-Room (R2R), Room-across-Room (RxR), Room-for-Room (R4R)) show the significant superiority of NavCoT over the direct action prediction variants. Through simple parameter-efficient finetuning, our NavCoT outperforms a recent GPT4-based approach with ~7% relative improvement on the R2R dataset. We believe that NavCoT will help unlock more task-adaptive and scalable LLM-based embodied agents, which are helpful for developing real-world robotics applications. Code is available at https://github.com/expectorlin/NavCoT.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 55 of its 68 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

55 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models26 May
  2. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
38 more
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  10. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  11. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  12. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  13. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
  14. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  15. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  16. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  17. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  18. 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
  19. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  20. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  21. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  22. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  23. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  24. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  25. 2019Stay on the Path: Instruction Fidelity in Vision-and-Language Navigationcited by 129
  26. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  27. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  28. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  29. 2019General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warpingcited by 102
  30. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  31. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  32. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  33. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigationcited by 71
  34. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  35. 2022HOP: History-and-Order Aware Pre-training for Vision-and-Language Navigationcited by 69
  36. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  37. 2022ADAPT: Vision-Language Navigation with Modality-Aligned Action Promptscited by 45
  38. 2019Counterfactual Vision-and-Language Navigation via Adversarial Path Samplingcited by 22

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 40; 2026: 27.

77 papers.

Sort:

By area:

  1. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  2. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  3. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  4. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 60
  5. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  6. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  7. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  8. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  9. 2025OctoNav: Towards Generalist Embodied Navigationcited by 33
  10. 2025UAV-Flow Colosseo: A Real-World Benchmark for Flying-on-a-Word UAV Imitation Learningcited by 30
  11. 2024RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigationcited by 23
  12. 2025RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoningcited by 19
38 more
  1. 2024Exploring Spatial Representation to Enhance LLM Reasoning in Aerial Vision-Language Navigationcited by 18
  2. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  3. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  4. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  5. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 15
  6. 2025SocialNav: Training Human-Inspired Foundation Model for Socially-Aware Embodied Navigationcited by 12
  7. 2025Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresightcited by 11
  8. 2025SORT3D: Spatial Object-centric Reasoning Toolbox for Zero-Shot 3D Grounding Using Large Language Modelscited by 10
  9. 2025Extending Large Vision-Language Model for Diverse Interactive Tasks in Autonomous Drivingcited by 10
  10. 2026Language-Conditioned World Modeling for Visual Navigationcited by 9
  11. 2025Nav-R2 Dual-Relation Reasoning for Generalizable Open-Vocabulary Object-Goal Navigationcited by 8
  12. 2026SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigationcited by 8
  13. 2024Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigationcited by 7
  14. 2025PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assemblycited by 7
  15. 2025HA-VLN 2.0: An Open Benchmark and Leaderboard for Human-Aware Navigation in Discrete and Continuous Environments with Dynamic Multi-Human Interactionscited by 6
  16. 2025Learning to Tune Like an Expert: Interpretable and Scene-Aware Navigation via MLLM Reasoning and CVAE-Based Adaptationcited by 6
  17. 2025VISTAv2: World Imagination for Indoor Vision-and-Language Navigationcited by 6
  18. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoningcited by 6
  19. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  20. 2025IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigationcited by 5
  21. 2025D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigationcited by 5
  22. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agentscited by 5
  23. 2026Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RLcited by 5
  24. 2026AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agentscited by 5
  25. 2026FutureNav: Unified World-Action Modeling for Vision-and-Language Navigationcited by 5
  26. 2026ABot-N1: Toward a General Visual Language Navigation Foundation Modelcited by 5
  27. 2025EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoningcited by 4
  28. 2025Generating Vision-Language Navigation Instructions Incorporated Fine-Grained Alignment Annotationscited by 4
  29. 2025Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigationcited by 4
  30. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agentscited by 4
  31. 2025MSNav: Zero-Shot Vision-and-Language Navigation with Dynamic Memory and LLM Spatial Reasoningcited by 4
  32. 2025How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptationcited by 4
  33. 2026NavWM: A Unified Navigation World Model for Foresight-Driven Planningcited by 4
  34. 2025Think Hierarchically, Act Dynamically: Hierarchical Multi-modal Fusion and Reasoning for Vision-and-Language Navigationcited by 3
  35. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  36. 2025Embodied Navigation with Auxiliary Task of Action Description Predictioncited by 3
  37. 2025UNeMo: Collaborative Visual-Language Reasoning and Navigation via a Multimodal World Modelcited by 3
  38. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chains
  2. 2026FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-Language Navigation
  3. 2025Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation
  4. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs
  5. 2025Vision-and-Language Navigation with Analogical Textual Descriptions in LLMs
  6. 2025Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation
  7. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning
  8. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation