mlsanityData to 7 Oct 2026

CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation

arXiv 2211.16649 ↗First posted 30 Nov 2022cs.CV cs.AI cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

Household environments are visually diverse. Embodied agents performing Vision-and-Language Navigation (VLN) in the wild must be able to handle this diversity, while also following arbitrary language instructions. Recently, Vision-Language models like CLIP have shown great performance on the task of zero-shot object recognition. In this work, we ask if these models are also capable of zero-shot language grounding. In particular, we utilize CLIP to tackle the novel problem of zero-shot VLN using natural language referring expressions that describe target objects, in contrast to past work that used simple language templates describing object classes. We examine CLIP's capability in making sequential navigational decisions without any dataset-specific finetuning, and study how it influences the path that an agent takes. Our results on the coarse-grained instruction following task of REVERIE demonstrate the navigational capability of CLIP, surpassing the supervised baseline in terms of both success rate (SR) and success weighted by path length (SPL). More importantly, we quantitatively show that our CLIP-based zero-shot approach generalizes better to show consistent performance across environments when compared to SOTA, fully supervised learning approaches when evaluated via Relative Change in Success (RCS).

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 24 of its 27 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

24 of 27 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  8. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  9. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  10. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  11. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  12. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
12 more
  1. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  2. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  3. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  4. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  5. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 116
  6. 2020Topological Planning with Transformers for Vision-and-Language Navigationcited by 115
  7. 2021Structured Scene Memory for Vision-Language Navigationcited by 95
  8. 2018Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigationcited by 84
  9. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  10. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  11. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  12. 2019Transferable Representation Learning in Vision-and-Language Navigationcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 23; 2025: 14; 2026: 6.

55 papers.

Sort:

By area:

  1. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  2. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  3. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  4. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 146
  5. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 135
  6. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  7. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  8. 2023VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street Viewcited by 67
  9. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  10. 2023A2Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Modelscited by 61
  11. 2025VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuningcited by 61
  12. 2024Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMscited by 60
38 more
  1. 2023LangNav: Language as a Perceptual Representation for Navigationcited by 47
  2. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 30
  3. 2024An Interactive Agent Foundation Modelcited by 25
  4. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 20
  5. 2023Find What You Want: Learning Demand-conditioned Object Attribute Space for Demand-driven Navigationcited by 18
  6. 2024Exploring Spatial Representation to Enhance LLM Reasoning in Aerial Vision-Language Navigationcited by 18
  7. 2024Zero-shot Object Navigation with Vision-Language Models Reasoningcited by 18
  8. 2025FlexVLN: Flexible Adaptation for Diverse Vision-and-Language Navigation Taskscited by 16
  9. 2023Advances in Embodied Navigation Using Large Language Models: A Surveycited by 14
  10. 2025SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Modelscited by 12
  11. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  12. 2025RAVEN: Resilient Aerial Navigation via Open-Set Semantic Memory and Behavior Adaptationcited by 10
  13. 2025CitySeeker: How Do VLMS Explore Embodied Urban Navigation With Implicit Human Needs?cited by 8
  14. 2023Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Trainingcited by 7
  15. 2024Semantic Shield: Defending Vision-Language Models Against Backdooring and Poisoning via Fine-grained Knowledge Alignmentcited by 7
  16. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Modelscited by 7
  17. 2024Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesiscited by 6
  18. 2024Semantic Map-based Generation of Navigation Instructionscited by 6
  19. 2024Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agentscited by 5
  20. 2025EfficientNav: Towards On-Device Object-Goal Navigation with Navigation Map Caching and Retrievalcited by 5
  21. 2024Visual Grounding for Object-Level Generalization in Reinforcement Learningcited by 4
  22. 2024Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigationcited by 4
  23. 2025History-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigationcited by 4
  24. 2026Three-Step Nav: A Hierarchical Global-Local Planner for Zero-Shot Vision-and-Language Navigationcited by 4
  25. 2024Personalized Embodied Navigation for Portable Object Findingcited by 3
  26. 2025Image-based Geo-localization for Robotics: Are Black-box Vision-Language Models there yet?cited by 3
  27. 2025TRAVEL: Training-Free Retrieval and Alignment for Vision-and-Language Navigationcited by 3
  28. 2025A-TPT: Angular Diversity Calibration Properties for Test-Time Prompt Tuning of Vision-Language Modelscited by 3
  29. 2023Labeling Indoor Scenes with Fusion of Out-of-the-Box Perception Modelscited by 2
  30. 2024Position Paper: Agent AI Towards a Holistic Intelligencecited by 2
  31. 2024Hierarchical Spatial Proximity Reasoning for Vision-and-Language Navigationcited by 2
  32. 2024One-Stage Open-Vocabulary Temporal Action Detection Leveraging Temporal Multi-scale and Action Label Featurescited by 2
  33. 2026POINav: Benchmarking and Enhancing Final-Meters Arrival in Real-World Vision-Language Navigationcited by 2
  34. 2026RAVEN: Long-Horizon Reasoning & Navigation with a Visuo-Spatio-Temporal Memorycited by 2
  35. 2023CARTIER: Cartographic lAnguage Reasoning Targeted at Instruction Execution for Robotscited by 1
  36. 2025Meme Similarity and Emotion Detection using Multimodal Analysiscited by 1
  37. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1
  38. 2024Simultaneous Localization and Affordance Prediction of Tasks from Egocentric Videocited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Models
  2. 2026ViTL: Temporal Logic-Guided Zero-Shot Natural Language Navigation via Vision-Language Models
  3. 2026AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness
  4. 2025Boosting Zero-Shot VLN via Abstract Obstacle Map-Based Waypoint Prediction with TopoGraph-and-VisitInfo-Aware Prompting
  5. 2024TINA: Think, Interaction, and Action Framework for Zero-Shot Vision Language Navigation
  6. 2025LaViRA: Language-Vision-Robot Actions Translation for Zero-Shot Vision Language Navigation in Continuous Environments
  7. 2026SFCo-Nav: Efficient Zero-Shot Visual Language Navigation via Collaboration of Slow LLM and Fast Attributed Graph Alignment
  8. 2025PanoNav: Mapless Zero-Shot Object Navigation with Panoramic Scene Parsing and Dynamic Memory