mlsanityData to 7 Oct 2026

The Surprising Effectiveness of Representation Learning for Visual Imitation

arXiv 2112.01511 ↗First posted 2 Dec 2021cs.RO cs.AI cs.CVcited by 149

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

While visual imitation learning offers one of the most effective ways of learning from visual demonstrations, generalizing from them requires either hundreds of diverse demonstrations, task specific priors, or large, hard-to-train parametric models. One reason such complexities arise is because standard visual imitation frameworks try to solve two coupled problems at once: learning a succinct but good representation from the diverse visual data, while simultaneously learning to associate the demonstrated actions with such representations. Such joint learning causes an interdependence between these two problems, which often results in needing large amounts of demonstrations for learning. To address this challenge, we instead propose to decouple representation learning from behavior learning for visual imitation. First, we learn a visual representation encoder from offline data using standard supervised and self-supervised learning methods. Once the representations are trained, we use non-parametric Locally Weighted Regression to predict the actions. We experimentally show that this simple decoupling improves the performance of visual imitation models on both offline demonstration datasets and real-robot door opening compared to prior work in visual imitation. All of our generated data, code, and robot videos are publicly available at https://jyopari.github.io/VINN/.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 53 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

36 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Playful Interactions for Representation Learning19 Jul
  2. 2020Visual Imitation Made Easy11 Aug
  3. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skills26 Feb

Other references · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2017Billion-scale similarity search with GPUscited by 2,477
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  8. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  11. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  12. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
21 more
  1. 2018Behavioral Cloning from Observationcited by 482
  2. 2021Implicit Behavioral Cloningcited by 442
  3. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  4. 2021With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representationscited by 324
  5. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  6. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  7. 2017One-Shot Imitation Learningcited by 232
  8. 2019SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learningcited by 194
  9. 2021Reinforcement Learning with Prototypical Representationscited by 179
  10. 2017A Survey of Structure from Motioncited by 157
  11. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  12. 2017Third-Person Imitation Learningcited by 103
  13. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 69
  14. 2017Towards Generalization and Simplicity in Continuous Controlcited by 65
  15. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  16. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  17. 2021The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environmentscited by 39
  18. 2018Q-learning with Nearest Neighborscited by 37
  19. 2020Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigationcited by 31
  20. 2019DoorGym: A Scalable Door Opening Environment And Baseline Agentcited by 26
  21. 2019Learning to Navigate Using Mid-Level Visual Priorscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 38; 2024: 42; 2025: 31; 2026: 19.

149 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  5. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  6. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  7. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  8. 2023A Cookbook of Self-Supervised Learningcited by 248
  9. 2022Masked Visual Pre-training for Motor Controlcited by 246
  10. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  11. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  12. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
38 more
  1. 2023Language-Driven Representation Learning for Roboticscited by 163
  2. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  3. 2022Human-to-Robot Imitation in the Wildcited by 157
  4. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 144
  5. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  6. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  7. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  8. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  9. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  10. 2023On Bringing Robots Homecited by 101
  11. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  12. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  13. 2023LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learningcited by 78
  14. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  15. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  16. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  17. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  18. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  19. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  20. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  21. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  22. 2023Waypoint-Based Imitation Learning for Robotic Manipulationcited by 59
  23. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  24. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  25. 2022Holo-Dex: Teaching Dexterity with Immersive Mixed Realitycited by 52
  26. 2023Train Offline, Test Online: A Real Robot Learning Benchmarkcited by 47
  27. 2025AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Controlcited by 47
  28. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  29. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  30. 2024DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Modelscited by 43
  31. 2024Robot Utility Models: General Policies for Zero-Shot Deployment in New Environmentscited by 43
  32. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  33. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learningcited by 42
  34. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  35. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 38
  36. 2024Surgical Robot Transformer (SRT): Imitation Learning for Surgical Taskscited by 38
  37. 2024ForceMimic: Force-Centric Imitation Learning with Force-Motion Capture System for Contact-Rich Manipulationcited by 36
  38. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 36

50 most cited of 149 All 149 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022An Empirical Investigation of Representation Learning for Imitation
  2. 2026Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
  3. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration
  4. 2021Manipulator-Independent Representations for Visual Imitation
  5. 2023Generalizable Imitation Learning Through Pre-Trained Representations
  6. 2021Towards More Generalizable One-shot Visual Imitation Learning
  7. 2023Robust Visual Imitation Learning with Inverse Dynamics Representations
  8. 2022Resolving Copycat Problems in Visual Imitation Learning via Residual Action Prediction