mlsanityData to 7 Oct 2026

BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning

arXiv 2202.02005 ↗First posted 4 Feb 2022cs.RO cs.LGcited by 517

On the map: World models → Embodied and robot world models

also in 1 more
Abstract (arXiv)

In this paper, we study the problem of enabling a vision-based robotic manipulation system to generalize to novel tasks, a long-standing challenge in robot learning. We approach the challenge from an imitation learning perspective, aiming to study how scaling and broadening the data collected can facilitate such generalization. To that end, we develop an interactive and flexible imitation learning system that can learn from both demonstrations and interventions and can be conditioned on different forms of information that convey the task, including pre-trained embeddings of natural language or videos of humans performing the task. When scaling data collection on a real robot to more than 100 distinct tasks, we find that this system can perform 24 unseen manipulation tasks with an average success rate of 44%, without any robot demonstrations for those tasks.

Key facts. Cited by 517 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 41 of its 56 references that are in the map. First posted on arXiv 4 Feb 2022.

Builds on

41 of 56 references.

References · 41

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  6. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  7. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  8. 2019Causal Confusion in Imitation Learningcited by 286
  9. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  10. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  11. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  12. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
29 more
  1. 2017One-Shot Imitation Learningcited by 232
  2. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  3. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  4. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  5. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  6. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  7. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  8. 2019Active Domain Randomizationcited by 161
  9. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  10. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  11. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  12. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  13. 2018Zero-Shot Visual Imitationcited by 114
  14. 2020Visual Imitation Made Easycited by 111
  15. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  16. 2020Transformers for One-Shot Visual Imitationcited by 77
  17. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  18. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 68
  19. 2018Task-Embedded Control Networks for Few-Shot Imitation Learningcited by 65
  20. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  21. 2018Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstrationcited by 60
  22. 2017Domain Randomization and Generative Models for Robotic Graspingcited by 34
  23. 2018VR-Goggles for Robots: Real-to-sim Domain Adaptation for Visual Controlcited by 34
  24. 2019Learning One-Shot Imitation from Humans without Humanscited by 32
  25. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Rewardcited by 26
  26. 2017Sim2Real View Invariant Visual Servoing by Recurrent Controlcited by 19
  27. 2021Zero-shot Task Adaptation using Natural Languagecited by 15
  28. 2020Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Datacited by 13
  29. 2018One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RLcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 38; 2023: 104; 2024: 137; 2025: 144; 2026: 94.

517 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  6. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  7. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  8. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  11. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  12. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
38 more
  1. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  3. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  4. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  5. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  6. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  7. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  8. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  9. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  10. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  11. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  12. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  13. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  14. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  15. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  16. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  17. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  18. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  19. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  20. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  21. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  22. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  23. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  24. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  25. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  26. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  27. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  28. 2022Multi-Game Decision Transformerscited by 212
  29. 2024RT-H: Action Hierarchies Using Languagecited by 204
  30. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 189
  31. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  32. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  33. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  34. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  35. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  36. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  37. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  38. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166

50 most cited of 517 All 517 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Towards More Generalizable One-shot Visual Imitation Learning
  2. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
  3. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation
  4. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos
  5. 2023Generalizable Imitation Learning Through Pre-Trained Representations
  6. 2025Generalization Capability for Imitation Learning
  7. 2025Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions
  8. 2025Learning a Thousand Tasks in a Day