mlsanityData to 7 Oct 2026

Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs

arXiv 2409.18794 ↗First posted 27 Sep 2024cs.RO cs.CVcited by 60

Not on the map. Why?

Abstract (arXiv)

Vision-and-Language Navigation (VLN) tasks require an agent to follow textual instructions to navigate through 3D environments. Traditional approaches use supervised learning methods, relying heavily on domain-specific datasets to train VLN models. Recent methods try to utilize closed-source large language models (LLMs) like GPT-4 to solve VLN tasks in zero-shot manners, but face challenges related to expensive token costs and potential data breaches in real-world applications. In this work, we introduce Open-Nav, a novel study that explores open-source LLMs for zero-shot VLN in the continuous environment. Open-Nav employs a spatial-temporal chain-of-thought (CoT) reasoning approach to break down tasks into instruction comprehension, progress estimation, and decision-making. It enhances scene perceptions with fine-grained object and spatial knowledge to improve LLM's reasoning in navigation. Our extensive experiments in both simulated and real-world environments demonstrate that Open-Nav achieves competitive performance compared to using closed-source LLMs.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 43 of its 48 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

43 of 48 references.

References · 43

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  9. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  10. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  11. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  12. 2023Recognize Anything: A Strong Image Tagging Modelcited by 305
31 more
  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  3. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  4. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  5. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  6. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  7. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  8. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  9. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  10. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  11. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  12. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  13. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  14. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  15. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  16. 2021Waypoint Models for Instruction-guided Navigation in Continuous Environmentscited by 125
  17. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  18. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  19. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  20. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  21. 2019The Regretful Agent: Heuristic-Aided Navigation through Progress Estimationcited by 104
  22. 2020Language and Visual Entity Relationship Graph for Agent Navigationcited by 100
  23. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  24. 2020Object-and-Action Aware Model for Visual Language Navigationcited by 77
  25. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  26. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  27. 2021Neighbor-view Enhanced Model for Vision and Language Navigationcited by 50
  28. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 47
  29. 2023KERM: Knowledge Enhanced Reasoning for Vision-and-Language Navigationcited by 38
  30. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  31. 2024MC-GPT: Empowering Vision-and-Language Navigation with Memory Map and Reasoning Chainscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 20; 2026: 38.

60 papers.

Sort:

By area:

  1. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  2. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  3. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  4. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  5. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  6. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  7. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  8. 2025DreamNav: A Trajectory-Based Imaginative Framework for Zero-Shot Vision-and-Language Navigationcited by 15
  9. 2025VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigationcited by 8
  10. 2025PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memorycited by 8
  11. 2025Nav-R2 Dual-Relation Reasoning for Generalizable Open-Vocabulary Object-Goal Navigationcited by 8
  12. 2026SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigationcited by 8
38 more
  1. 2026HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory Systemcited by 8
  2. 2025Active Test-time Vision-Language Navigationcited by 7
  3. 2025STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimizationcited by 6
  4. 2025Fast-SmartWay: Panoramic-Free End-to-End Zero-Shot Vision-and-Language Navigationcited by 6
  5. 2025UrbanNav: Learning Language-Guided Urban Navigation from Web-Scale Human Trajectoriescited by 6
  6. 2026Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigationcited by 6
  7. 2026AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harnesscited by 6
  8. 2026LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigationcited by 5
  9. 2026One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representationcited by 5
  10. 2025Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environmentscited by 4
  11. 2025Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Promptingcited by 4
  12. 2026Spatial-VLN: Zero-Shot Vision-and-Language Navigation With Explicit Spatial Perception and Explorationcited by 4
  13. 2026Three-Step Nav: A Hierarchical Global-Local Planner for Zero-Shot Vision-and-Language Navigationcited by 4
  14. 20263DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splattingcited by 3
  15. 2026Structured Observation Language for Efficient and Generalizable Vision-Language Navigationcited by 3
  16. 2026Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agentscited by 3
  17. 2026Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigationcited by 3
  18. 2026SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoningcited by 3
  19. 2026HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harnesscited by 3
  20. 2025HyPerNav: Hybrid Perception for Object-Oriented Navigation in Unknown Environmentcited by 2
  21. 2025UrbanVLA: A Vision-Language-Action Model for Urban Micromobilitycited by 2
  22. 2026SpatialAnt: Autonomous Zero-Shot Robot Navigation via Active Scene Reconstruction and Visual Anticipationcited by 2
  23. 2026P2DNav: Panorama-to-Downview Reasoning for Zero-shot Vision-and-Language Navigationcited by 2
  24. 2026Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigationcited by 2
  25. 2026CLOSER-VLN: Closed-Loop Self-Verified Retrieval-Augmented Reasoning for Aerial Vision-Language Navigationcited by 2
  26. 2025User-Feedback-Driven Adaptation for Vision-and-Language Navigationcited by 1
  27. 2026All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptationcited by 1
  28. 2026LightZeroNav: Zero-Shot Vision Language Navigation in Continuous Environments Based on Lightweight VLMscited by 1
  29. 2026FLYNN: Robust Neural Network for Robot Navigation using Fly Brain Topologycited by 1
  30. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1
  31. 2026Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memorycited by 1
  32. 2026HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigationcited by 1
  33. 2025NVSim: Novel View Synthesis Simulator for Large Scale Indoor Navigationcited by 0
  34. 2025TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Makingcited by 0
  35. 2026PM-Nav: Priori-Map Guided Embodied Navigation in Functional Buildingscited by 0
  36. 2026What Limits Vision-and-Language Navigation ?cited by 0
  37. 2026Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactionscited by 0
  38. 2026VISTA: Scale-Aware Visual Navigation via Action History Conditioningcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models
  2. 2026C2Nav: Compare Before You Commit for Zero-Shot Vision-and-Language Navigation
  3. 2024TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigation
  4. 2026OpenFrontier: General Navigation with Visual-Language Grounded Frontiers
  5. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models
  6. 2025OpenNav: Open-World Navigation with Multimodal Large Language Models
  7. 2026Exploring Bottlenecks in VLM-LLM Navigation: How 3D Scene Understanding Capability Impacts Zero-Shot VLN
  8. 2026SFCo-Nav: Efficient Zero-Shot Visual Language Navigation via Collaboration of Slow LLM and Fast Attributed Graph Alignment