mlsanityData to 7 Oct 2026

Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?

arXiv 2204.11134 ↗First posted 23 Apr 2022cs.RO cs.AIcited by 70

Not on the map. Why?

Abstract (arXiv)

Task specification is at the core of programming autonomous robots. A low-effort modality for task specification is critical for engagement of non-expert end-users and ultimate adoption of personalized robot agents. A widely studied approach to task specification is through goals, using either compact state vectors or goal images from the same robot scene. The former is hard to interpret for non-experts and necessitates detailed state estimation and scene understanding. The latter requires the generation of desired goal image, which often requires a human to complete the task, defeating the purpose of having autonomous robots. In this work, we explore alternate and more general forms of goal specification that are expected to be easier for humans to specify and use such as images obtained from the internet, hand sketches that provide a visual description of the desired task, or simple language descriptions. As a preliminary step towards this, we investigate the capabilities of large scale pre-trained models (foundation models) for zero-shot goal specification, and find promising results in a collection of simulated robot manipulation tasks and real-world datasets.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 23 Apr 2022.

Builds on

37 of 46 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
25 more
  1. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  3. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  4. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 435
  5. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  6. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  8. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  9. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  10. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  11. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  12. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  13. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  14. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  15. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  16. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  17. 2020The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Posecited by 95
  18. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  19. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  20. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  21. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  22. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  23. 2019IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Taskscited by 68
  24. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  25. 2017Towards Generalization and Simplicity in Continuous Controlcited by 65

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 20; 2024: 23; 2025: 12; 2026: 8.

70 papers.

Sort:

By area:

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  4. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  5. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  6. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  7. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  8. 2023Language-Driven Representation Learning for Roboticscited by 163
  9. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  10. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  11. 2023Vision-Language Models as Success Detectorscited by 138
  12. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
38 more
  1. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  2. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  3. 2023Vision-Language Models as a Source of Rewardscited by 65
  4. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  5. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  6. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  7. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  8. 2022Retrospectives on the Embodied AI Workshopcited by 46
  9. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  10. 2024RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketchescited by 41
  11. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  12. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  13. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 38
  14. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  15. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 32
  16. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 29
  17. 2023CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a 10,000 Budget; An Extra 4,000 Unlocks 81.8% Accuracycited by 23
  18. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  19. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  20. 2023Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?cited by 21
  21. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 19
  22. 2024Exploring Spatial Representation to Enhance LLM Reasoning in Aerial Vision-Language Navigationcited by 18
  23. 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17
  24. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  25. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  26. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14
  27. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13
  28. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  29. 2022A System for Morphology-Task Generalization via Unified Representation and Behavior Distillationcited by 12
  30. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  31. 2025BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Modelscited by 10
  32. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 10
  33. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Modelscited by 9
  34. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8
  35. 2023DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physicscited by 6
  36. 2024Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learningcited by 6
  37. 2025Chain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Modelscited by 6
  38. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5

50 most cited of 70 All 70 (JSON)