mlsanityData to 7 Oct 2026

The Unsurprising Effectiveness of Pre-Trained Vision Models for Control

arXiv 2203.03580 ↗First posted 7 Mar 2022cs.CV cs.AI cs.LGcited by 155

Not on the map. Why?

Abstract (arXiv)

Recent years have seen the emergence of pre-trained representations as a powerful abstraction for AI applications in computer vision, natural language, and speech. However, policy learning for control is still dominated by a tabula-rasa learning paradigm, with visuo-motor policies often trained from scratch using data from deployment environments. In this context, we revisit and study the role of pre-trained visual representations for control, and in particular representations trained on large-scale computer vision datasets. Through extensive empirical evaluation in diverse control domains (Habitat, DeepMind Control, Adroit, Franka Kitchen), we isolate and study the importance of different representation training methods, data augmentations, and feature hierarchies. Overall, we find that pre-trained visual representations can be competitive or even better than ground-truth state representations to train control policies. This is in spite of using only out-of-domain data from standard vision datasets, without any in-domain data from the deployment environments. Source code and more at https://sites.google.com/view/pvr-control.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 7 Mar 2022.

Builds on

57 of 74 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2017Mask R-CNNcited by 7,259
  10. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
38 more
  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2015Fast R-CNNcited by 2,930
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  5. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  11. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  12. 2018DeepMind Control Suitecited by 935
  13. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  14. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  15. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  16. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  17. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  18. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  19. 2018Learning Dexterous In-Hand Manipulationcited by 643
  20. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  21. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  22. 2020Reinforcement Learning with Augmented Datacited by 452
  23. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  24. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  25. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  26. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  27. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  28. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  29. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  30. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  31. 2021Reinforcement Learning with Prototypical Representationscited by 179
  32. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  33. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 159
  34. 2020Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biasescited by 143
  35. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  36. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  37. 2021Visual Room Rearrangementcited by 101
  38. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 23; 2023: 61; 2024: 41; 2025: 24; 2026: 6.

155 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2022A Generalist Agentcited by 802
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  6. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  7. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2023Reward Design with Language Modelscited by 257
  10. 2023A Cookbook of Self-Supervised Learningcited by 248
  11. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  12. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
38 more
  1. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  2. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  3. 2022Masked World Models for Visual Controlcited by 169
  4. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  5. 2023Language-Driven Representation Learning for Roboticscited by 163
  6. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  7. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  8. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  9. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  10. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  11. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  12. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  13. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 106
  14. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  15. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  16. 2024Foundation Policies with Hilbert Representationscited by 89
  17. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  18. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  19. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  20. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  21. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  22. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  23. 2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 56
  24. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  25. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  26. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  27. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  28. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  29. 2023Train Offline, Test Online: A Real Robot Learning Benchmarkcited by 47
  30. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  31. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  32. 2024Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learningcited by 45
  33. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  34. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  35. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  36. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  37. 2023Learning Navigational Visual Representations with Semantic Map Supervisioncited by 40
  38. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40

50 most cited of 155 All 155 (JSON)