mlsanityData to 14 Oct 2026

VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation

arXiv 2312.03275 ↗First posted 6 Dec 2023cs.RO cs.AIcited by 205

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Understanding how humans leverage semantic knowledge to navigate unfamiliar environments and decide where to explore next is pivotal for developing robots capable of human-like search behaviors. We introduce a zero-shot navigation approach, Vision-Language Frontier Maps (VLFM), which is inspired by human reasoning and designed to navigate towards unseen semantic objects in novel environments. VLFM builds occupancy maps from depth observations to identify frontiers, and leverages RGB observations and a pre-trained vision-language model to generate a language-grounded value map. VLFM then uses this map to identify the most promising frontier to explore for finding an instance of a given target object category. We evaluate VLFM in photo-realistic environments from the Gibson, Habitat-Matterport 3D (HM3D), and Matterport 3D (MP3D) datasets within the Habitat simulator. Remarkably, VLFM achieves state-of-the-art results on all three datasets as measured by success weighted by path length (SPL) for the Object Goal Navigation task. Furthermore, we show that VLFM's zero-shot nature enables it to be readily deployed on real-world robots such as the Boston Dynamics Spot mobile manipulation platform. We deploy VLFM on Spot and demonstrate its capability to efficiently navigate to target objects within an office building in the real world, without any prior knowledge of the environment. The accomplishments of VLFM underscore the promising potential of vision-language models in advancing the field of semantic navigation. Videos of real-world deployment can be viewed at naoki.io/vlfm.

Key facts. Cited by 205 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 38 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

34 of 38 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  8. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
  9. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  11. 2018On Evaluation of Embodied Navigation Agentscited by 547
  12. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
22 more
  1. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  2. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  3. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  4. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  5. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  6. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 178
  7. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 150
  8. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 127
  9. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 93
  10. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  11. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 70
  12. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60
  13. 2023How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontierscited by 54
  14. 20223D-Aware Object Goal Navigation via Simultaneous Exploration and Identificationcited by 46
  15. 2022Is Mapping Necessary for Realistic PointGoal Navigation?cited by 35
  16. 2021Hierarchical Object-to-Zone Graph for Object Navigationcited by 34
  17. 2022Stubborn: A Strong Baseline for Indoor Object Navigationcited by 32
  18. 2022Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigationcited by 30
  19. 2021Success Weighted by Completion Time: A Dynamics-Aware Evaluation Criteria for Embodied Navigationcited by 22
  20. 2022VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangementcited by 18
  21. 2022Object Goal Navigation using Data Regularized Q-Learningcited by 10
  22. 2022Learning-Augmented Model-Based Planning for Visual Explorationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 36; 2025: 75; 2026: 94.

205 papers.

Sort:

By area:

  1. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 157
  2. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 109
  3. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  4. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 75
  5. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  6. 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
  7. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 67
  8. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 57
  9. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51
  10. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 50
  11. 2025NavDP: Learning Sim-to-Real Navigation Diffusion Policy with Privileged Information Guidancecited by 49
  12. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
38 more
  1. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  2. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 43
  3. 2024GraspSplats: Efficient Manipulation with 3D Feature Splattingcited by 41
  4. 2024Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigationcited by 39
  5. 2024GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answeringcited by 37
  6. 2024DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulationcited by 32
  7. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  8. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  9. 2024TANGO: Training-free Embodied AI Agents for Open-world Taskscited by 27
  10. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  11. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 25
  12. 2026ABot-N0: Technical Report on the VLA Foundation Model for Versatile Embodied Navigationcited by 25
  13. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  14. 2025RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Explorationcited by 23
  15. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 22
  16. 2025Grounded Vision-Language Navigation for UAVs with Open-Vocabulary Goal Understandingcited by 19
  17. 2025Describe Anything Anywhere At Any Momentcited by 19
  18. 2024SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environmentscited by 18
  19. 2025AstraNav-World: World Model for Foresight Control and Consistencycited by 17
  20. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  21. 2024EfficientEQA: An Efficient Approach to Open-Vocabulary Embodied Question Answeringcited by 15
  22. 2025Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answeringcited by 14
  23. 2024One Map to Find Them All: Real-time Open-Vocabulary Mapping for Zero-shot Multi-Object Navigationcited by 13
  24. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 13
  25. 2025ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigationcited by 13
  26. 2025MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigationcited by 13
  27. 2025FrontierNet: Learning Visual Cues to Explorecited by 12
  28. 2025Aerial World Model for Long-horizon Visual Generation and Navigation in 3D Spacecited by 12
  29. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 11
  30. 2025Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environmentscited by 10
  31. 2025Mobi-π: Mobilizing Your Robot Learning Policycited by 10
  32. 2025RAVEN: Resilient Aerial Navigation via Open-Set Semantic Memory and Behavior Adaptationcited by 10
  33. 2025Nav-R2 Dual-Relation Reasoning for Generalizable Open-Vocabulary Object-Goal Navigationcited by 10
  34. 2026VLingNav: Embodied Navigation with Adaptive Reasoning and Visual-Assisted Linguistic Memorycited by 10
  35. 2026Sparse Video Generation Propels Real-World Beyond-the-View Vision-Language Navigationcited by 10
  36. 2024Robot Navigation Using Physically Grounded Vision-Language Models in Outdoor Environmentscited by 9
  37. 2024Multimodal LLM Guided Exploration and Active Mapping using Fisher Informationcited by 9
  38. 2025ForesightNav: Learning Scene Imagination for Efficient Explorationcited by 9

50 most cited of 205 All 205 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VLN-Game: Vision-Language Equilibrium Search for Zero-Shot Semantic Navigation
  2. 2025BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation
  3. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
  4. 2024IVLMap: Instance-Aware Visual Language Grounding for Consumer Robot Navigation
  5. 2025FlySearch: Exploring how vision-language models explore
  6. 2026Instance-Enriched Semantic Maps for Visual Language Navigation
  7. 2025MAG-Nav: Language-Driven Object Navigation Leveraging Memory-Reserved Active Grounding ↗
  8. 2025Think, Remember, Navigate: Zero-Shot Object-Goal Navigation with VLM-Powered Reasoning