mlsanityData to 7 Oct 2026

Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning

arXiv 2212.08860 ↗First posted 17 Dec 2022cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

Learning generalizable policies that can adapt to unseen environments remains challenging in visual Reinforcement Learning (RL). Existing approaches try to acquire a robust representation via diversifying the appearances of in-domain observations for better generalization. Limited by the specific observations of the environment, these methods ignore the possibility of exploring diverse real-world image datasets. In this paper, we investigate how a visual RL agent would benefit from the off-the-shelf visual representations. Surprisingly, we find that the early layers in an ImageNet pre-trained ResNet model could provide rather generalizable representations for visual RL. Hence, we propose Pre-trained Image Encoder for Generalizable visual reinforcement learning (PIE-G), a simple yet effective framework that can generalize to the unseen visual scenarios in a zero-shot manner. Extensive experiments are conducted on DMControl Generalization Benchmark, DMControl Manipulation Tasks, Drawer World, and CARLA to verify the effectiveness of PIE-G. Empirical evidence suggests PIE-G improves sample efficiency and significantly outperforms previous state-of-the-art methods in terms of generalization performance. In particular, PIE-G boasts a 55% generalization performance gain on average in the challenging video background setting. Project Page: https://sites.google.com/view/pie-g/home.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 74 of its 88 references that are in the map. First posted on arXiv 17 Dec 2022.

Builds on

74 of 88 references.

References · 74

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
38 more
  1. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2018DeepMind Control Suitecited by 935
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  12. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  13. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  14. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  15. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  16. 2020Reinforcement Learning with Augmented Datacited by 452
  17. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  18. 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 422
  19. 2016Learning to reinforcement learncited by 369
  20. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  21. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  22. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  23. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  24. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  25. 2019Self-Supervised Exploration via Disagreementcited by 263
  26. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  27. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  28. 2022Masked Visual Pre-training for Motor Controlcited by 246
  29. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  30. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  31. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  32. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  33. 2021Reinforcement Learning with Prototypical Representationscited by 179
  34. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  35. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  36. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  37. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  38. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 20; 2025: 12; 2026: 5.

58 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  3. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  4. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  5. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  6. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  7. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  8. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  9. 2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
  10. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  11. 2024UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AIcited by 25
  12. 2024Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasetscited by 23
38 more
  1. 2023For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equalcited by 22
  2. 2024PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Controlcited by 21
  3. 2024Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RLcited by 21
  4. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
  5. 2024Generative Dataset Distillation: Balancing Global Structure and Local Detailscited by 18
  6. 2024End-to-End Neuro-Symbolic Reinforcement Learning with Textual Explanationscited by 17
  7. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  8. 2023A Universal Semantic-Geometric Representation for Robotic Manipulationcited by 16
  9. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  10. 2023Normalization Enhances Generalization in Visual Reinforcement Learningcited by 13
  11. 2023Improving Generalization in Visual Reinforcement Learning via Conflict-aware Gradient Agreement Augmentationcited by 13
  12. 2023Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generationcited by 13
  13. 2024Leveraging Locality to Boost Sample Efficiency in Robotic Manipulationcited by 13
  14. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  15. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12
  16. 2024Premier-TACO is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Losscited by 10
  17. 2023MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learningcited by 9
  18. 2025Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learningcited by 8
  19. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 7
  20. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learningcited by 7
  21. 2023Visual Encoders for Data-Efficient Imitation Learning in Modern Video Gamescited by 6
  22. 2024DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delayscited by 6
  23. 2023PASTA: Pretrained Action-State Transformer Agentscited by 5
  24. 2024Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidencecited by 5
  25. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  26. 2023Learning Generalizable Agents via Saliency-Guided Features Decorrelationcited by 3
  27. 2025Modeling Unseen Environments with Language-guided Composable Causal Components in Reinforcement Learningcited by 3
  28. 2024Pretrained Visual Representations in Reinforcement Learningcited by 2
  29. 2025Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encodercited by 2
  30. 2025Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learningcited by 2
  31. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  32. 2024Efficient Training of Generalizable Visuomotor Policies via Control-Aware Augmentationcited by 1
  33. 2025Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learningcited by 1
  34. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Datacited by 1
  35. 2026PriorZero: Bridging Language Priors and World Models for Decision Makingcited by 1
  36. 2023Value Explicit Pretraining for Learning Transferable Representationscited by 0
  37. 2024Adapting Pretrained ViTs with Convolution Injector for Visuo-Motor Controlcited by 0
  38. 2024An Examination of Offline-Trained Encoders in Vision-Based Deep Reinforcement Learning for Autonomous Drivingcited by 0

50 most cited of 58 All 58 (JSON)