mlsanityData to 7 Oct 2026

Three-Step Nav: A Hierarchical Global-Local Planner for Zero-Shot Vision-and-Language Navigation

arXiv 2604.26946 ↗First posted 29 Apr 2026cs.CV cs.ROcited by 4

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Breakthrough progress in vision-based navigation through unknown environments has been achieved by using multimodal large language models (MLLMs). These models can plan a sequence of motions by evaluating the current view at each time step against the task and goal given to the agent. However, current zero-shot Vision-and-Language Navigation (VLN) agents powered by MLLMs still tend to drift off course, halt prematurely, and achieve low overall success rates. We propose Three-Step Nav to counteract these failures with a three-view protocol: First, "look forward" to extract global landmarks and sketch a coarse plan. Then, "look now" to align the current visual observation with the next sub-goal for fine-grained guidance. Finally, "look backward" audits the entire trajectory to correct accumulated drift before stopping. Requiring no gradient updates or task-specific fine-tuning, our planner drops into existing VLN pipelines with minimal overhead. Three-Step Nav achieves state-of-the-art zero-shot performance on the R2R-CE and RxR-CE dataset. Our code is available at https://github.com/ZoeyZheng0/3-step-Nav.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 32 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

23 of 32 references.

References · 23

  1. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  2. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  3. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  4. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  5. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  6. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  7. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  8. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  9. 2022Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigationcited by 127
  10. 2022BEVBert: Multimodal Map Pre-training for Language-guided Navigationcited by 117
  11. 2023GridMM: Grid Memory Map for Vision-and-Language Navigationcited by 117
  12. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
11 more
  1. 2023Scaling Data Generation in Vision-and-Language Navigationcited by 115
  2. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  3. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  4. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  5. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 60
  6. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
  7. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigationcited by 55
  8. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  9. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  10. 2025SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigationcited by 29
  11. 2024VLN-Video: Utilizing Driving Videos for Outdoor Vision-and-Language Navigationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigationcited by 3
  2. 2026One MLLM, One Call: Efficient Zero-Shot Vision-and-Language Navigation via Spatial-Aware Waypointscited by 0
  3. 2026GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMscited by 0
  4. 2026Affordance-Conditioned Decision Making: Bridging the Semantic-Spatial Gap in Zero-Shot Cross-Floor Vision-and-Language Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026C2Nav: Compare Before You Commit for Zero-Shot Vision-and-Language Navigation
  2. 2026AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness
  3. 2026ViTL: Temporal Logic-Guided Zero-Shot Natural Language Navigation via Vision-Language Models
  4. 2025FiLM-Nav: Efficient and Generalizable Navigation via VLM Fine-tuning
  5. 2025VLN-Zero: Rapid Exploration and Cache-Enabled Neurosymbolic Vision-Language Planning for Zero-Shot Transfer in Robot Navigation
  6. 2026CATNAV: Cached Vision-Language Traversability for Efficient Zero-Shot Robot Navigation
  7. 2026SFCo-Nav: Efficient Zero-Shot Visual Language Navigation via Collaboration of Slow LLM and Fast Attributed Graph Alignment
  8. 2026SEDualVLN: A Spatially-Enhanced Dual-System for Vision-Language Navigation