mlsanityData to 14 Oct 2026

Pretrained Visual Representations in Reinforcement Learning

arXiv 2407.17238 ↗First posted 24 Jul 2024cs.RO cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Visual reinforcement learning (RL) has made significant progress in recent years, but the choice of visual feature extractor remains a crucial design decision. This paper compares the performance of RL algorithms that train a convolutional neural network (CNN) from scratch with those that utilize pre-trained visual representations (PVRs). We evaluate the Dormant Ratio Minimization (DRM) algorithm, a state-of-the-art visual RL method, against three PVRs: ResNet18, DINOv2, and Visual Cortex (VC). We use the Metaworld Push-v2 and Drawer-Open-v2 tasks for our comparison. Our results show that the choice of training from scratch compared to using PVRs for maximising performance is task-dependent, but PVRs offer advantages in terms of reduced replay buffer size and faster training times. We also identify a strong correlation between the dormant ratio and model performance, highlighting the importance of exploration in visual RL. Our study provides insights into the trade-offs between training from scratch and using PVRs, informing the design of future visual RL algorithms.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 17 of its 17 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

17 of 17 references.

References · 17

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  6. 2023Vision Transformers Need Registerscited by 871
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  9. 2022Masked Visual Pre-training for Motor Controlcited by 248
  10. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  11. 2023Language-Driven Representation Learning for Roboticscited by 165
  12. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
5 more
  1. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 107
  2. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 82
  3. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 78
  4. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  5. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  2. 2025Pre-trained Visual Representations Generalize Where it Matters in Model-Based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
  2. 2024Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learning
  3. 2024When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?
  4. 2024R3L: Relative Representations for Reinforcement Learning
  5. 2025Efficient Reinforcement Learning Through Adaptively Pretrained Visual Encoder
  6. 2025Reinforcement Learning for Large Model: A Survey
  7. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control
  8. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning