mlsanityData to 7 Oct 2026

Simple but Effective: CLIP Embeddings for Embodied AI

arXiv 2111.09888 ↗First posted 18 Nov 2021cs.CV cs.LGcited by 191

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive language image pretraining (CLIP) encoders have been shown to be beneficial for a range of visual tasks from classification and detection to captioning and image manipulation. We investigate the effectiveness of CLIP visual backbones for Embodied AI tasks. We build incredibly simple baselines, named EmbCLIP, with no task specific architectures, inductive biases (such as the use of semantic maps), auxiliary tasks during training, or depth maps -- yet we find that our improved baselines perform very well across a range of tasks and simulators. EmbCLIP tops the RoboTHOR ObjectNav leaderboard by a huge margin of 20 pts (Success Rate). It tops the iTHOR 1-Phase Rearrangement leaderboard, beating the next best submission, which employs Active Neural Mapping, and more than doubling the % Fixed Strict metric (0.08 to 0.17). It also beats the winners of the 2021 Habitat ObjectNav Challenge, which employ auxiliary tasks, depth maps, and human demonstrations, and those of the 2019 Habitat PointNav Challenge. We evaluate the ability of CLIP's visual representations at capturing semantic information about input observations -- primitives that are useful for navigation-heavy embodied tasks -- and find that CLIP's representations encode these primitives more effectively than ImageNet-pretrained backbones. Finally, we extend one of our baselines, producing an agent capable of zero-shot object navigation that can navigate to objects that were not used as targets during training. Our code and models are available at https://github.com/allenai/embodied-clip

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Builds on 32 of its 39 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

32 of 39 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  5. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  8. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  11. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  12. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
20 more
  1. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  2. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  3. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  4. 2020Learning to Explore using Active Neural SLAMcited by 308
  5. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  6. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  7. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  8. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  9. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  10. 2020Rearrangement: A Challenge for Embodied AIcited by 153
  11. 2021Visual Room Rearrangementcited by 101
  12. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  13. 2021Auxiliary Tasks and Exploration Enable ObjectNavcited by 73
  14. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  15. 2020AllenAct: A Framework for Embodied AI Researchcited by 54
  16. 2021RobustNav: Towards Benchmarking Robustness in Embodied Navigationcited by 52
  17. 2019Two Body Problem: Collaborative Visual Task Completioncited by 45
  18. 2020Bridging the Imitation Gap by Adaptive Insubordinationcited by 38
  19. 2021GridToPix: Training Embodied Agents with Minimal Supervisioncited by 20
  20. 2020How to Train PointGoal Navigation Agents on a (Sample and Compute) Budgetcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 42; 2023: 63; 2024: 48; 2025: 27; 2026: 11.

191 papers.

Sort:

By area:

  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  5. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  6. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  7. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  10. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  11. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  12. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
38 more
  1. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  2. 2023Language-Driven Representation Learning for Roboticscited by 163
  3. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  4. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  5. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  6. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  7. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  8. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  9. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  10. 2022Navigating to Objects in the Real Worldcited by 104
  11. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 98
  12. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  13. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  14. 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
  15. 2023CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIPcited by 89
  16. 2024C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersioncited by 82
  17. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  18. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  19. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  20. 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
  21. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  22. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69
  23. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  24. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  25. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 58
  26. 2024VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Modelcited by 55
  27. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  28. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  29. 2023SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environmentscited by 53
  30. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  31. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  32. 2022Retrospectives on the Embodied AI Workshopcited by 46
  33. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  34. 2024Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learningcited by 45
  35. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  36. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  37. 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 42
  38. 2022NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fieldscited by 41

50 most cited of 191 All 191 (JSON)