mlsanityData to 14 Oct 2026

End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering

arXiv 2411.05755 ↗First posted 8 Nov 2024cs.RO cs.CL cs.CVcited by 37

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

We present VLMnav, an embodied framework to transform a Vision-Language Model (VLM) into an end-to-end navigation policy. In contrast to prior work, we do not rely on a separation between perception, planning, and control; instead, we use a VLM to directly select actions in one step. Surprisingly, we find that a VLM can be used as an end-to-end policy zero-shot, i.e., without any fine-tuning or exposure to navigation data. This makes our approach open-ended and generalizable to any downstream navigation task. We run an extensive study to evaluate the performance of our approach in comparison to baseline prompting methods. In addition, we perform a design analysis to understand the most impactful design decisions. Visual examples and code for our project can be found at https://jirl-upenn.github.io/VLMnav/

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 39 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

28 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models26 May

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  9. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  10. 2018On Evaluation of Embodied Navigation Agentscited by 547
  11. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  12. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
15 more
  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  2. 2023ViNT: A Foundation Model for Visual Navigationcited by 222
  3. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  4. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  5. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  6. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  7. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  8. 2022Habitat-Matterport 3D Semantics Datasetcited by 110
  9. 2023GOAT: GO to Any Thingcited by 89
  10. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  11. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 67
  12. 2024Does Spatial Cognition Emerge in Frontier Models?cited by 66
  13. 2024VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Taskscited by 47
  14. 2024CoNVOI: Context-aware Navigation using Vision Language Models in Outdoor and Indoor Environmentscited by 43
  15. 2018Frontier Based Exploration for Autonomous Robotcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 18.

37 papers.

Sort:

By area:

  1. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 43
  2. 2025Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learningcited by 41
  3. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  4. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  5. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  6. 2025NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Taskscited by 10
  7. 2025WoMAP: World Models For Embodied Open-Vocabulary Object Localizationcited by 9
  8. 2025CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?cited by 8
  9. 2025CogDDN: A Cognitive Demand-Driven Navigation with Decision Optimization and Dual-Process Thinkingcited by 7
  10. 2025DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigationcited by 6
  11. 2026CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor Navigationcited by 6
  12. 2025Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Explorationcited by 5
25 more
  1. 2026Allocentric Perceiver: Disentangling Allocentric Reasoning from Egocentric Visual Priors via Frame Instantiationcited by 5
  2. 2026GSMem: 3D Gaussian Splatting as Persistent Spatial Memory for Zero-Shot Embodied Exploration and Reasoningcited by 5
  3. 2026The Dual Mechanisms of Spatial Variable Binding in Vision-Language Modelscited by 4
  4. 2024Preference Goal Tuning: Post-Training as Latent Control for Frozen Policiescited by 3
  5. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  6. 2025SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimizationcited by 3
  7. 2026OpenFrontier: General Navigation with Visual-Language Grounded Frontierscited by 3
  8. 2026Structured Observation Language for Efficient and Generalizable Vision-Language Navigationcited by 3
  9. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  10. 2025SoraNav: Adaptive UAV Task-Centric Navigation via Zeroshot VLM Reasoningcited by 2
  11. 2025LISN: Language-Instructed Social Navigation with VLM-based Controller Modulatingcited by 2
  12. 2026MerNav: A Highly Generalizable Memory-Execute-Review Framework for Zero-Shot Object Goal Navigationcited by 2
  13. 2026CoFL: Continuous Flow Fields for Language-Conditioned Navigationcited by 2
  14. 2026ReMemNav: Memory-Based Decision Correction and Target Verification for Zero-Shot Object Navigationcited by 2
  15. 2026WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Modelscited by 2
  16. 2026Visually-grounded Humanoid Agentscited by 2
  17. 2026LIME: Learning Intent-aware Camera Motion from Egocentric Videocited by 2
  18. 2025ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imaginationcited by 1
  19. 2026EvoMemNav: Efficient Self-Evolving Fine-Grained Memory for Zero-Shot Embodied Navigationcited by 1
  20. 2025When Engineering Outruns Intelligence: Rethinking Instruction-Guided Navigationcited by 0
  21. 2025TP-MDDN: Task-Preferenced Multi-Demand-Driven Navigation with Autonomous Decision-Makingcited by 0
  22. 2026Semantic-Aware Guided Drone Exploration for Language-Conditioned 3D Indoor Mappingcited by 0
  23. 2026VTM-Nav: Harnessing Cross-Episode Experience for Object-Goal Navigation with Hierarchical Visual-Topological Memorycited by 0
  24. 2026Map the Possibilities: Spatial Belief Fields for Language-Goal Aerial Navigationcited by 0
  25. 2026Beyond Spatial Benchmarks: From Spatial Reasoning to Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  2. 2025Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation
  3. 2026What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
  4. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions
  5. 2025A Navigation Framework Utilizing Vision-Language Models
  6. 2025Think, Remember, Navigate: Zero-Shot Object-Goal Navigation with VLM-Powered Reasoning
  7. 2026AgentVLN: Towards Agentic Vision-and-Language Navigation
  8. 2026Embodied-Navigator: Point, Think, Memorize, and Align for Efficient Navigation