mlsanityData to 7 Oct 2026

Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation

arXiv 2307.03659 ↗First posted 7 Jul 2023cs.RO cs.AIcited by 78

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

What makes generalization hard for imitation learning in visual robotic manipulation? This question is difficult to approach at face value, but the environment from the perspective of a robot can often be decomposed into enumerable factors of variation, such as the lighting conditions or the placement of the camera. Empirically, generalization to some of these factors have presented a greater obstacle than others, but existing work sheds little light on precisely how much each factor contributes to the generalization gap. Towards an answer to this question, we study imitation learning policies in simulation and on a real robot language-conditioned manipulation task to quantify the difficulty of generalization to different (sets of) factors. We also design a new simulated benchmark of 19 tasks with 11 factors of variation to facilitate more controlled evaluations of generalization. From our study, we determine an ordering of factors based on generalization difficulty, that is consistent across simulation and our real robot setup.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 57 references that are in the map. First posted on arXiv 7 Jul 2023.

Builds on

47 of 57 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training30 Sep
  2. 2020Visual Imitation Made Easy11 Aug
  3. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitation7 Nov

Other references · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
32 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  4. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  5. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2018Universal Sentence Encodercited by 485
  8. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  9. 2020Reinforcement Learning with Augmented Datacited by 452
  10. 2017Robust Adversarial Reinforcement Learningcited by 439
  11. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  13. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  14. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  15. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  16. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  17. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  18. 2021Partial success in closing the gap between human and machine visioncited by 185
  19. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  20. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  21. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  22. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  23. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  24. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  25. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  26. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  27. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  28. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  29. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  30. 2023Train Offline, Test Online: A Real Robot Learning Benchmarkcited by 47
  31. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  32. 2022Learning Dense Visual Descriptors using Image Augmentations for Robot Manipulation Taskscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 24; 2025: 32; 2026: 19.

78 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  3. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  4. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  5. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  6. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  7. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  8. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  9. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 56
  10. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  11. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  12. 2025Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulationcited by 42
38 more
  1. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  2. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  3. 2024Efficient Data Collection for Robotic Manipulation via Compositional Generalizationcited by 35
  4. 2024View-Invariant Policy Learning via Zero-Shot Novel View Synthesiscited by 35
  5. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  6. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
  7. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  8. 2024Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Trainingcited by 26
  9. 2025GeoAware-VLA: Implicit Geometry Aware Vision-Language-Action Modelcited by 24
  10. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Modelscited by 23
  11. 2025What Matters in Learning from Large-Scale Datasets for Robot Manipulationcited by 20
  12. 2025Refined Policy Distillation: From VLA Generalists to RL Expertscited by 19
  13. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitationcited by 18
  14. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 16
  15. 2025StereoVLA: Enhancing Vision-Language-Action Models with Stereo Visioncited by 16
  16. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  17. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 15
  18. 2025A Taxonomy for Evaluating Generalist Robot Manipulation Policiescited by 15
  19. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
  20. 2024Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulationcited by 11
  21. 2024M2Distill: Multi-Modal Distillation for Lifelong Imitation Learningcited by 10
  22. 2025Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learningcited by 9
  23. 2024Contrast Sets for Evaluating Language-Guided Robot Policiescited by 7
  24. 2025Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selectioncited by 7
  25. 2025BOSS: Benchmark for Observation Space Shift in Long-Horizon Taskcited by 7
  26. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 7
  27. 2025VO-DP: Semantic-Geometric Adaptive Diffusion Policy for Vision-Only Robotic Manipulationcited by 7
  28. 2026AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Modelscited by 7
  29. 2025Object-Centric Representations Improve Policy Generalization in Robot Manipulationcited by 6
  30. 2025How Do VLAs Effectively Inherit from VLMs?cited by 6
  31. 2024RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fieldscited by 5
  32. 2025EveryDayVLA: A Vision-Language-Action Model for Affordable Robotic Manipulationcited by 5
  33. 2024Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaborationcited by 4
  34. 2024Stem-OB: Generalizable Visual Imitation Learning with Stem-Like Convergent Observation through Diffusion Inversioncited by 4
  35. 2025eFlesh: Highly customizable Magnetic Touch Sensing using Cut-Cell Microstructurescited by 4
  36. 2025The Temporal Trap: Entanglement in Pre-Trained Visual Representations for Visuomotor Policy Learningcited by 3
  37. 2025CIVIL: Causal and Intuitive Visual Imitation Learningcited by 3
  38. 2026HeRO: Hierarchical 3D Semantic Representation for Pose-aware Object Manipulationcited by 3

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalization Capability for Imitation Learning
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
  3. 2021Towards More Generalizable One-shot Visual Imitation Learning
  4. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
  5. 2020Generalization Guarantees for Imitation Learning
  6. 2023Generalizable Imitation Learning Through Pre-Trained Representations
  7. 2026Inductive Generalization for Robotic Manipulation
  8. 2026An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning