mlsanityData to 7 Oct 2026

OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models

arXiv 2402.10670 ↗First posted 16 Feb 2024cs.CL cs.ROcited by 66

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Object navigation (ObjectNav) requires an agent to navigate through unseen environments to find queried objects. Many previous methods attempted to solve this task by relying on supervised or reinforcement learning, where they are trained on limited household datasets with close-set objects. However, two key challenges are unsolved: understanding free-form natural language instructions that demand open-set objects, and generalizing to new environments in a zero-shot manner. Aiming to solve the two challenges, in this paper, we propose OpenFMNav, an Open-set Foundation Model based framework for zero-shot object Navigation. We first unleash the reasoning abilities of large language models (LLMs) to extract proposed objects from natural language instructions that meet the user's demand. We then leverage the generalizability of large vision language models (VLMs) to actively discover and detect candidate objects from the scene, building a Versatile Semantic Score Map (VSSM). Then, by conducting common sense reasoning on VSSM, our method can perform effective language-guided exploration and exploitation of the scene and finally reach the goal. By leveraging the reasoning and generalizing abilities of foundation models, our method can understand free-form human instructions and perform effective open-set zero-shot navigation in diverse environments. Extensive experiments on the HM3D ObjectNav benchmark show that our method surpasses all the strong baselines on all metrics, proving our method's effectiveness. Furthermore, we perform real robot demonstrations to validate our method's open-set-ness and generalizability to real-world environments.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 50 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

38 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation6 Dec

Other references · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Segment Anythingcited by 8,158
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2021Grounded Language-Image Pre-trainingcited by 1,147
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
25 more
  1. 2022A Survey on In-context Learningcited by 717
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  3. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  4. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  5. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  6. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  7. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  8. 2018Semi-parametric Topological Memory for Navigationcited by 203
  9. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  10. 2020Neural Topological SLAM for Visual Navigationcited by 193
  11. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  12. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  13. 2023L3MVN: Leveraging Large Language Models for Visual Target Navigationcited by 118
  14. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  15. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  16. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
  17. 2022Navigating to Objects in the Real Worldcited by 104
  18. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  19. 2023Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skillcited by 75
  20. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  21. 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 65
  22. 20223D-Aware Object Goal Navigation via Simultaneous Exploration and Identificationcited by 46
  23. 2020Semantic Curiosity for Active Visual Learningcited by 44
  24. 2023Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigationcited by 18
  25. 2023Object Goal Navigation with Recursive Implicit Mapscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 26; 2026: 26.

66 papers.

Sort:

By area:

  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  2. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 146
  3. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  4. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  5. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  6. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 46
  7. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 42
  8. 2024ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imaginationcited by 38
  9. 2024End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answeringcited by 37
  10. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  11. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  12. 2025ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmarkcited by 18
38 more
  1. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  2. 2024Enhancing Multi-Robot Semantic Navigation Through Multimodal Chain-of-Thought Score Collaborationcited by 14
  3. 2025CL-CoTNav: Closed-Loop Hierarchical Chain-of-Thought for Zero-Shot Object-Goal Navigation with Vision-Language Modelscited by 11
  4. 2025GC-VLN: Instruction as Graph Constraints for Training-free Vision-and-Language Navigationcited by 10
  5. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 10
  6. 2025SkillBlender: Towards Versatile Humanoid Whole-Body Loco-Manipulation via Skill Blendingcited by 9
  7. 2024Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialoguescited by 7
  8. 2025FlySearch: Exploring how vision-language models explorecited by 7
  9. 2025DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigationcited by 6
  10. 2025STRIVE: Structured Representation Integrating VLM Reasoning for Efficient Object Navigationcited by 5
  11. 2026LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigationcited by 5
  12. 2024Find Everything: A General Vision Language Model Approach to Multi-Object Searchcited by 4
  13. 2024MLLM-Search: A Zero-Shot Approach to Finding People using Multimodal Large Language Modelscited by 4
  14. 2025RATE-Nav: Region-Aware Termination Enhancement for Zero-shot Object Navigation with Vision-Language Modelscited by 4
  15. 2025AdaptBot: Combining LLM with Knowledge Graphs and Human Input for Generic-to-Specific Task Decomposition and Knowledge Refinementcited by 3
  16. 2025VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perceptioncited by 3
  17. 20263DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splattingcited by 3
  18. 2026Stop Wandering: Efficient Vision-Language Navigation via Metacognitive Reasoningcited by 3
  19. 2026Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigationcited by 3
  20. 2024OpenObject-NAV: Open-Vocabulary Object-Oriented Navigation Based on Dynamic Carrier-Relationship Scene Graphcited by 2
  21. 2024Navigation with VLM framework: Towards Going to Any Languagecited by 2
  22. 2025RANa: Retrieval-Augmented Navigationcited by 2
  23. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Modelscited by 2
  24. 2025HyPerNav: Hybrid Perception for Object-Oriented Navigation in Unknown Environmentcited by 2
  25. 2026ReMemNav: Memory-Based Decision Correction and Target Verification for Zero-Shot Object Navigationcited by 2
  26. 2026PSG-Nav: Probabilistic Scene Graph Navigation via Multiverse Decision Makingcited by 2
  27. 2025Embodied Crowd Countingcited by 1
  28. 2025Toward Ownership Understanding of Objects: Active Question Generation with Large Language Model and Probabilistic Generative Modelcited by 1
  29. 2025MUVLA: Learning to Explore Object Navigation via Map Understandingcited by 1
  30. 2025ImagineNav++: Prompting Vision-Language Models as Embodied Navigator through Scene Imaginationcited by 1
  31. 2025Schr"odinger's Navigator: Imagining an Ensemble of Futures for Zero-Shot Object Navigationcited by 1
  32. 2026MerNav: A Highly Generalizable Memory-Execute-Review Framework for Zero-Shot Object Goal Navigationcited by 1
  33. 2026ReasonNavi: Human-Inspired Global Map Reasoning for Zero-Shot Embodied Navigationcited by 1
  34. 2026AERR-Nav: Adaptive Exploration-Recovery-Reminiscing Strategy for Zero-Shot Object Navigationcited by 1
  35. 2026Memory Over Maps: 3D Object Localization Without Reconstructioncited by 1
  36. 2026ConsistNav: Closing the Action Consistency Gap in Zero-Shot Object Navigation with Semantic Executive Controlcited by 1
  37. 2026IntentionNav: A Benchmark for Intent-Driven Object Navigation from Implicit Human Instructioncited by 1
  38. 2026MVP-Nav: Multi-layer Value Map Planner Navigatorcited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs
  2. 2025osmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation via Semantic Maps and Large Language Models Reasoning
  3. 2022CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation
  4. 2025OpenNav: Open-World Navigation with Multimodal Large Language Models
  5. 2024TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigation
  6. 2025History-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigation
  7. 2026ViTL: Temporal Logic-Guided Zero-Shot Natural Language Navigation via Vision-Language Models
  8. 2026HRO: Hierarchical Room-to-Object Framework for Zero-Shot Object Goal Navigation with Large Language Models