mlsanityData to 14 Oct 2026

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings

arXiv 2206.12403 ↗First posted 24 Jun 2022cs.CV cs.LG cs.ROcited by 178

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a scalable approach for learning open-world object-goal navigation (ObjectNav) -- the task of asking a virtual robot (agent) to find any instance of an object in an unexplored environment (e.g., "find a sink"). Our approach is entirely zero-shot -- i.e., it does not require ObjectNav rewards or demonstrations of any kind. Instead, we train on the image-goal navigation (ImageNav) task, in which agents find the location where a picture (i.e., goal image) was captured. Specifically, we encode goal images into a multimodal, semantic embedding space to enable training semantic-goal navigation (SemanticNav) agents at scale in unannotated 3D environments (e.g., HM3D). After training, SemanticNav agents can be instructed to find objects described in free-form natural language (e.g., "sink", "bathroom sink", etc.) by projecting language goals into the same multimodal, semantic embedding space. As a result, our approach enables open-world ObjectNav. We extensively evaluate our agents on three ObjectNav datasets (Gibson, HM3D, and MP3D) and observe absolute improvements in success of 4.2% - 20.0% over existing zero-shot methods. For reference, these gains are similar or better than the 5% improvement in success between the Habitat 2020 and 2021 ObjectNav challenge winners. In an open-world setting, we discover that our agents can generalize to compound instructions with a room explicitly mentioned (e.g., "Find a kitchen sink") and when the target room can be inferred (e.g., "Find a sink and a stove").

Key facts. Cited by 178 papers in the map (data through 14 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 24 Jun 2022.

Builds on

31 of 52 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  12. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
19 more
  1. 2018On Evaluation of Embodied Navigation Agentscited by 547
  2. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  3. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  4. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
  5. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  6. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 362
  7. 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
  8. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  9. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 244
  10. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  11. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  12. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 117
  13. 2022Offline Visual Representation Learning for Embodied Navigationcited by 75
  14. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 67
  15. 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 66
  16. 2020SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigationcited by 44
  17. 2022Stubborn: A Strong Baseline for Indoor Object Navigationcited by 32
  18. 2021Semi-Supervised Vision Transformerscited by 24
  19. 2020BenchBot: Evaluating Robotics Research in Photorealistic 3D Simulation and on Real Robotscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 29; 2024: 50; 2025: 52; 2026: 40.

178 papers.

Sort:

By area:

  1. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  2. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  3. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  4. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 205
  5. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  6. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 157
  7. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 127
  8. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 109
  9. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 94
  10. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 89
  11. 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 75
  12. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
38 more
  1. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 73
  2. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 70
  3. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 67
  4. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  5. 2024Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigationcited by 65
  6. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 62
  7. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 58
  8. 2024VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Modelcited by 57
  9. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 57
  10. 2022Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instancescited by 55
  11. 2023Navigating to Objects Specified by Imagescited by 51
  12. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 51
  13. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 50
  14. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  15. 2022Retrospectives on the Embodied AI Workshopcited by 46
  16. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 43
  17. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  18. 2024CityWalker: Learning Embodied Urban Navigation from Web-Scale Videoscited by 41
  19. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  20. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 33
  21. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 31
  22. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  23. 2024Prioritized Semantic Learning for Zero-shot Instance Navigationcited by 31
  24. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  25. 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
  26. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 27
  27. 2022Phone2Proc: Bringing Robust Robots Into Our Chaotic Worldcited by 26
  28. 2025Prior Does Matter: Visual Navigation via Denoising Diffusion Bridge Modelscited by 25
  29. 2023FGPrompt: Fine-grained Goal Prompting for Image-goal Navigationcited by 22
  30. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 22
  31. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Promptingcited by 20
  32. 2024Zero-shot Object Navigation with Vision-Language Models Reasoningcited by 19
  33. 2023Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigationcited by 18
  34. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 17
  35. 2023Object Goal Navigation with Recursive Implicit Mapscited by 16
  36. 2023End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenoncited by 16
  37. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 16
  38. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16

50 most cited of 178 All 178 (JSON)