Generalizable Imitation Learning Through Pre-Trained Representations
On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning
Abstract (arXiv)
In this paper, we leverage self-supervised vision transformer models and their emergent semantic abilities to improve the generalization abilities of imitation learning policies. We introduce DVK, an imitation learning algorithm that leverages rich pre-trained Visual Transformer patch-level embeddings to obtain better generalization when learning through demonstrations. Our learner sees the world by clustering appearance features into groups associated with semantic concepts, forming stable keypoints that generalize across a wide range of appearance variations and object types. We demonstrate how this representation enables generalized behaviour by evaluating imitation learning across a diverse dataset of object manipulation tasks. To facilitate further study of generalization in Imitation Learning, all of our code for the method and evaluation, as well as the dataset, is made available.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 56 references that are in the map. First posted on arXiv 15 Nov 2023.
Builds on
44 of 56 references.
Closest ancestors · 2
Same direction, newest first
- 2021Generalization Through Hand-Eye Coordination: An Action Space for Learning Spatially-Invariant Visuomotor Control28 Feb
- 2019Self-Supervised Correspondence in Visuomotor Policy Learning16 Sep
Other references · 42
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
- 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
- 2015End-to-End Training of Deep Visuomotor Policiescited by 875
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
- 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
30 more
- 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
- 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
- 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 612
- 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
- 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
- 2021Implicit Behavioral Cloningcited by 452
- 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
- 2021Deep ViT Features as Dense Visual Descriptorscited by 330
- 2022Masked Visual Pre-training for Motor Controlcited by 248
- 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 241
- 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
- 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
- 2023Language-Driven Representation Learning for Roboticscited by 165
- 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
- 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 148
- 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 142
- 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
- 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
- 2022Better plain ViT baselines for ImageNet-1kcited by 98
- 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
- 2022Learning to Merge Tokens in Vision Transformerscited by 65
- 2017Deep Object-Centric Representations for Generalizable Robot Learningcited by 50
- 2019Graph-Structured Visual Imitationcited by 43
- 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 42
- 2018Deep Object-Centric Policies for Autonomous Drivingcited by 41
- 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 39
- 2021Gaze-based dual resolution deep imitation learning for high-precision dexterous robot manipulationcited by 17
- 2022IL-flOw: Imitation Learning from Observation using Normalizing Flowscited by 12
- 2023Imitation Learning from Observation through Optimal Transportcited by 7
- 2021Geometry-Aware Fruit Grasping Estimation for Robotic Harvesting in Orchardscited by 6
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Generalization Capability for Imitation Learning
- 2021Towards More Generalizable One-shot Visual Imitation Learning
- 2022Learning Category-Level Generalizable Object Manipulation Policy via Generative Adversarial Self-Imitation Learning from Demonstrations
- 2021The Surprising Effectiveness of Representation Learning for Visual Imitation
- 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation
- 2025Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions
- 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning
- 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations