mlsanityData to 7 Oct 2026

CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation

arXiv 2203.10421 ↗First posted 20 Mar 2022cs.CV cs.LG cs.ROcited by 194

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

For robots to be generally useful, they must be able to find arbitrary objects described by people (i.e., be language-driven) even without expensive navigation training on in-domain data (i.e., perform zero-shot inference). We explore these capabilities in a unified setting: language-driven zero-shot object navigation (L-ZSON). Inspired by the recent success of open-vocabulary models for image classification, we investigate a straightforward framework, CLIP on Wheels (CoW), to adapt open-vocabulary models to this task without fine-tuning. To better evaluate L-ZSON, we introduce the Pasture benchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects. We conduct an in-depth empirical study by directly deploying 21 CoW baselines across Habitat, RoboTHOR, and Pasture. In total, we evaluate over 90k navigation episodes and find that (1) CoW baselines often struggle to leverage language descriptions, but are proficient at finding uncommon objects. (2) A simple CoW, with CLIP-based object localization and classical exploration -- and no additional training -- matches the navigation efficiency of a state-of-the-art ZSON method trained for 500M steps on Habitat MP3D data. This same CoW provides a 15.6 percentage point improvement in success over a state-of-the-art RoboTHOR ZSON model.

Key facts. Cited by 194 papers in the map (data through 7 Oct 2026). Builds on 63 of its 90 references that are in the map. First posted on arXiv 20 Mar 2022.

Builds on

63 of 90 references.

References · 63

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  11. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
38 more
  1. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  3. 2018Exploration by Random Network Distillationcited by 966
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  6. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  7. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  8. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  9. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  10. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  11. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  12. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  13. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  14. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  15. 2020Learning to Explore using Active Neural SLAMcited by 308
  16. 2019Zero-Shot Semantic Segmentationcited by 245
  17. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  18. 2017Cognitive Mapping and Planning for Visual Navigationcited by 209
  19. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  20. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  21. 2018Zero-Shot Object Detectioncited by 189
  22. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  23. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149
  24. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  25. 2018Visual Semantic Navigation using Scene Priorscited by 127
  26. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 118
  27. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  28. 2019Learning Exploration Policies for Navigationcited by 113
  29. 2021Visual Room Rearrangementcited by 101
  30. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  31. 2018Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learningcited by 91
  32. 2018QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAMcited by 86
  33. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  34. 2017MINOS: Multimodal Indoor Simulator for Navigation in Complex Environmentscited by 66
  35. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 66
  36. 2021Memory-Augmented Reinforcement Learning for Image-Goal Navigationcited by 64
  37. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 63
  38. 2020Semantic Visual Navigation by Watching YouTube Videoscited by 60

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 25; 2024: 61; 2025: 56; 2026: 51.

194 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  3. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  4. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  5. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  6. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  7. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  8. 2023VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigationcited by 201
  9. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  10. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  11. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  12. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
38 more
  1. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  2. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  3. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  4. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  5. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  6. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  7. 2023Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigationcited by 74
  8. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  9. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  10. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  11. 2024OpenFMNav: Towards Open-Set Zero-Shot Object Navigation via Vision-Language Foundation Modelscited by 66
  12. 2023SATR: Zero-Shot Semantic Segmentation of 3D Shapescited by 62
  13. 2025UniGoal: Towards Universal Zero-shot Goal-oriented Navigationcited by 56
  14. 2024VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Modelcited by 55
  15. 2023Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representationcited by 54
  16. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  17. 2024GaussNav: Gaussian Splatting for Visual Navigationcited by 49
  18. 2024Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environmentscited by 49
  19. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  20. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 42
  21. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 41
  22. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  23. 2023Single-view 3D Scene Reconstruction with High-fidelity Shape and Texturecited by 37
  24. 2025OmniVLA: An Omni-Modal Vision-Language-Action Model for Robot Navigationcited by 34
  25. 2023Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphscited by 33
  26. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  27. 2024CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMscited by 32
  28. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  29. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  30. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Modelscited by 28
  31. 2024An Interactive Agent Foundation Modelcited by 25
  32. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  33. 2024Zero-shot Object Navigation with Vision-Language Models Reasoningcited by 18
  34. 2024g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Taskscited by 18
  35. 2024Correctable Landmark Discovery via Large Models for Vision-Language Navigationcited by 16
  36. 2024SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Expertscited by 16
  37. 2024Verifiably Following Complex Robot Instructions with Foundation Modelscited by 15
  38. 2024Instance-aware Exploration-Verification-Exploitation for Instance ImageGoal Navigationcited by 15

50 most cited of 194 All 194 (JSON)