mlsanityData to 14 Oct 2026

Open-World Object Manipulation using Pre-trained Vision-Language Models

arXiv 2303.00905 ↗First posted 2 Mar 2023cs.RO cs.AI cs.CVcited by 156

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary, e.g. "can you get me the pink stuffed whale?" to their sensory observations and actions. This brings up a notably difficult challenge for robots: while robot learning approaches allow robots to learn many different behaviors from first-hand experience, it is impractical for robots to have first-hand experiences that span all of this semantic information. We would like a robot's policy to be able to perceive and pick up the pink stuffed whale, even if it has never seen any data interacting with a stuffed whale before. Fortunately, static data on the internet has vast semantic information, and this information is captured in pre-trained vision-language models. In this paper, we study whether we can interface robot policies with these pre-trained models, with the aim of allowing robots to complete instructions involving object categories that the robot has never seen first-hand. We develop a simple approach, which we call Manipulation of Open-World Objects (MOO), which leverages a pre-trained vision-language model to extract object-identifying information from the language command and image, and conditions the robot policy on the current image, the instruction, and the extracted object information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments. In addition, we show how MOO generalizes to other, non-language-based input modalities to specify the object of interest such as finger pointing, and how it can be further extended to enable open-world navigation and manipulation. The project's website and evaluation videos can be found at https://robot-moo.github.io/

Key facts. Cited by 156 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 59 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

47 of 59 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Mask R-CNNcited by 7,277
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  7. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  9. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  11. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  12. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
35 more
  1. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  5. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  6. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  7. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  8. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  9. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  10. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  11. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  12. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  13. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  14. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
  15. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  16. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
  17. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  18. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 160
  19. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  20. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 138
  21. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  22. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 128
  23. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  24. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  25. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  26. 2020Visual Imitation Made Easycited by 111
  27. 2021RRL: Resnet as representation for Reinforcement Learningcited by 91
  28. 2021Greedy Hierarchical Variational Autoencoders for Large-Scale Video Predictioncited by 75
  29. 2019Does computer vision matter for action?cited by 64
  30. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  31. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 51
  32. 2016Robotic Grasp Detection using Deep Convolutional Neural Networkscited by 47
  33. 2021Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordancescited by 40
  34. 2020Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigationcited by 32
  35. 2019Differentiable Algorithm Networks for Composable Robot Learningcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 28; 2024: 49; 2025: 51; 2026: 28.

156 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  3. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  4. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  5. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  6. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  7. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 346
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  9. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  10. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
  11. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  12. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 138
38 more
  1. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 134
  2. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 134
  3. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  4. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  5. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  7. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 123
  8. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  9. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 108
  10. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 107
  11. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 106
  12. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  13. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  14. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  15. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  16. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75
  17. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 71
  18. 2024OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitationcited by 67
  19. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 61
  20. 2023Learning Generalizable Manipulation Policies with Object-Centric 3D Representationscited by 61
  21. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  22. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 59
  23. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 56
  24. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 55
  25. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  26. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 52
  27. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 52
  28. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 51
  29. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructionscited by 51
  30. 2024Vision-based Manipulation from Single Human Video with Open-World Object Graphscited by 50
  31. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  32. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  33. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 43
  34. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  35. 2024Diffusion Transformer Policycited by 41
  36. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 40
  37. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  38. 2023TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systemscited by 39

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Which objects help me to act effectively? Reasoning about physically-grounded affordances
  2. 2023Robotic Applications of Pre-Trained Vision-Language Models to Various Recognition Behaviors
  3. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation
  4. 2023Vision-Language Foundation Models as Effective Robot Imitators
  5. 2025KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulation
  6. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Models
  7. 2024Affordance Perception by a Knowledge-Guided Vision-Language Model with Efficient Error Correction
  8. 2025LaVA-Man: Learning Visual Action Representations for Robot Manipulation