mlsanityData to 7 Oct 2026

InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment

arXiv 2406.04882 ↗First posted 7 Jun 2024cs.RO cs.AI cs.CLcited by 150

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Enabling robots to navigate following diverse language instructions in unexplored environments is an attractive goal for human-robot interaction. However, this goal is challenging because different navigation tasks require different strategies. The scarcity of instruction navigation data hinders training an instruction navigation model with varied strategies. Therefore, previous methods are all constrained to one specific type of navigation instruction. In this work, we propose InstructNav, a generic instruction navigation system. InstructNav makes the first endeavor to handle various instruction navigation tasks without any navigation training or pre-built maps. To reach this goal, we introduce Dynamic Chain-of-Navigation (DCoN) to unify the planning process for different types of navigation instructions. Furthermore, we propose Multi-sourced Value Maps to model key elements in instruction navigation so that linguistic DCoN planning can be converted into robot actionable trajectories. With InstructNav, we complete the R2R-CE task in a zero-shot way for the first time and outperform many task-training methods. Besides, InstructNav also surpasses the previous SOTA method by 10.48% on the zero-shot Habitat ObjNav and by 86.34% on demand-driven navigation DDN. Real robot experiments on diverse indoor scenes further demonstrate our method's robustness in coping with the environment and instruction variations.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Builds on 40 of its 45 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

40 of 45 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  9. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  10. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  11. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  12. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
28 more
  1. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  2. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  3. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  4. 2022Visual Language Maps for Robot Navigationcited by 362
  5. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  6. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  7. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  8. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  9. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  10. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  11. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  12. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  13. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  14. 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 118
  15. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  16. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  17. 2023Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skillcited by 75
  18. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  19. 2020The Marathon 2: A Navigation Systemcited by 67
  20. 2024VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Modelcited by 55
  21. 2021SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigationcited by 53
  22. 2023How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontierscited by 53
  23. 2021SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environmentscited by 48
  24. 2023Amodal Ground Truth and Completion in the Wildcited by 43
  25. 2022A Tri-Layer Plugin to Improve Occluded Detectioncited by 21
  26. 2023Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigationcited by 18
  27. 2023From the Desks of ROS Maintainers: A Survey of Modern & Capable Mobile Robotics Algorithms in the Robot Operating System 2cited by 10
  28. 2022Predicting Dense and Context-aware Cost Maps for Semantic Robot Navigationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 51; 2026: 83.

150 papers.

Sort:

By area:

  1. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 146
  2. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  3. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Modelscited by 96
  4. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 80
  5. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  6. 2024Towards Long-Horizon Vision-Language Navigation: Platform, Benchmark and Methodcited by 66
  7. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  8. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  9. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  10. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  11. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 46
  12. 2025Openfly: A comprehensive platform for aerial vision-language navigationcited by 43
38 more
  1. 2024CityNav: A Large-Scale Dataset for Real-World Aerial Navigationcited by 40
  2. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  3. 2025OctoNav: Towards Generalist Embodied Navigationcited by 33
  4. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  5. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  6. 2025Beyond the Destination: A Novel Benchmark for Exploration-Aware Embodied Question Answeringcited by 29
  7. 2025CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Modelcited by 26
  8. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  9. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  10. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 24
  11. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 23
  12. 2025Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigationcited by 21
  13. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  14. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  15. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  16. 2025Cross from Left to Right Brain: Adaptive Text Dreamer for Vision-and-Language Navigationcited by 16
  17. 2025DreamNav: A Trajectory-Based Imaginative Framework for Zero-Shot Vision-and-Language Navigationcited by 15
  18. 2025MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robotscited by 15
  19. 2025NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Predictioncited by 15
  20. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  21. 2025Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answeringcited by 12
  22. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigationcited by 11
  23. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  24. 2026VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memorycited by 10
  25. 2026Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigationcited by 10
  26. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 10
  27. 2024Multimodal LLM Guided Exploration and Active Mapping using Fisher Informationcited by 9
  28. 2025When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answeringcited by 9
  29. 2025VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigationcited by 8
  30. 2025PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memorycited by 8
  31. 2026SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-and-Language Navigationcited by 8
  32. 2026HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory Systemcited by 8
  33. 2026AgentVLN: Towards Agentic Vision-and-Language Navigationcited by 8
  34. 2025FlySearch: Exploring how vision-language models explorecited by 7
  35. 2025Active Test-time Vision-Language Navigationcited by 7
  36. 2025CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulationcited by 7
  37. 2026Dual-Anchoring: Addressing State Drift in Vision-Language Navigationcited by 7
  38. 2024Pragmatic Embodied Spoken Instruction Following in Human-Robot Collaboration with Theory of Mindcited by 6

50 most cited of 150 All 150 (JSON)