mlsanityData to 7 Oct 2026

VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning

arXiv 2202.10324 ↗First posted 17 Feb 2022cs.CV cs.AI cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

We propose VRL3, a powerful data-driven framework with a simple design for solving challenging visual deep reinforcement learning (DRL) tasks. We analyze a number of major obstacles in taking a data-driven approach, and present a suite of design principles, novel findings, and critical insights about data-driven visual DRL. Our framework has three stages: in stage 1, we leverage non-RL datasets (e.g. ImageNet) to learn task-agnostic visual representations; in stage 2, we use offline RL data (e.g. a limited number of expert demonstrations) to convert the task-agnostic representations into more powerful task-specific representations; in stage 3, we fine-tune the agent with online RL. On a set of challenging hand manipulation tasks with sparse reward and realistic visual inputs, compared to the previous SOTA, VRL3 achieves an average of 780% better sample efficiency. And on the hardest task, VRL3 is 1220% more sample efficient (2440% when using a wider encoder) and solves the task with only 10% of the computation. These significant results clearly demonstrate the great potential of data-driven deep reinforcement learning.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 70 of its 113 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

70 of 113 references.

References · 70

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
38 more
  1. 2014How transferable are features in deep neural networks?cited by 1,216
  2. 2018DeepMind Control Suitecited by 935
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  9. 2017Deep Reinforcement Learning that Matterscited by 727
  10. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  13. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  14. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  15. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  16. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  17. 2020Reinforcement Learning with Augmented Datacited by 452
  18. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  19. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  20. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  21. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  22. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  23. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  24. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  25. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  26. 2019Fixing the train-test resolution discrepancycited by 209
  27. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  28. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  29. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  30. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  31. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  32. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  33. 2020Model-Based Offline Planningcited by 126
  34. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learningcited by 125
  35. 2020Phasic Policy Gradientcited by 121
  36. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  37. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
  38. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 16; 2024: 14; 2025: 10; 2026: 4.

51 papers.

Sort:

By area:

  1. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  2. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  3. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  4. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  5. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 56
  6. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  7. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  8. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  9. 2023Finetuning Offline World Models in the Real Worldcited by 39
  10. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  11. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  12. 2023Polybot: Training One Policy Across Robots While Embracing Variabilitycited by 28
38 more
  1. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  2. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  3. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  4. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  5. 2023For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equalcited by 22
  6. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
  7. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  8. 2025Dense Policy: Bidirectional Autoregressive Learning of Actionscited by 19
  9. 2023SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learningcited by 16
  10. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  11. 2025H3DP: Triply-Hierarchical Diffusion Policy for Visuomotor Learningcited by 13
  12. 2025FreqPolicy: Efficient Flow-based Visuomotor Policy via Frequency Consistencycited by 13
  13. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12
  14. 2024Score and Distribution Matching Policy: Advanced Accelerated Visuomotor Policies via Matched Distillationcited by 12
  15. 2023Pre-training with Synthetic Data Helps Offline Reinforcement Learningcited by 9
  16. 2024Visually Robust Adversarial Imitation Learning from Videos with Contrastive Learningcited by 9
  17. 2023Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learningcited by 8
  18. 2023CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observationscited by 8
  19. 2025Spatial-Temporal Aware Visuomotor Diffusion Policy Learningcited by 8
  20. 2022Aerial View Localization with Reinforcement Learning: Towards Emulating Search-and-Rescuecited by 4
  21. 2024Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learningcited by 4
  22. 2025Improve the Training Efficiency of DRL for Wireless Communication Resource Allocation: The Role of Generative Diffusion Modelscited by 4
  23. 2024MOOSS: Mask-Enhanced Temporal Contrastive Learning for Smooth State Evolution in Visual Reinforcement Learningcited by 3
  24. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  25. 2025Sample from What You See: Visuomotor Policy Learning via Diffusion Bridge with Observation-Embedded Stochastic Differential Equationcited by 2
  26. 2026X-Distill: Cross-Architecture Vision Distillation for Visuomotor Learningcited by 2
  27. 2023Boosting Reinforcement Learning and Planning with Demonstrations: A Surveycited by 1
  28. 2023H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulationcited by 1
  29. 2024Efficient Training of Generalizable Visuomotor Policies via Control-Aware Augmentationcited by 1
  30. 2024Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Plannercited by 1
  31. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Datacited by 1
  32. 2025Disentangled Object-Centric Image Representation for Robotic Manipulationcited by 1
  33. 2025Beyond Domain Randomization: Event-Inspired Perception for Visually Robust Adversarial Imitation from Videoscited by 1
  34. 2023Policy-Induced Self-Supervision Improves Representation Finetuning in Visual RLcited by 0
  35. 2024Adapting Pretrained ViTs with Convolution Injector for Visuo-Motor Controlcited by 0
  36. 2025ISS Policy : Scalable Diffusion Policy with Implicit Scene Supervisioncited by 0
  37. 2026Primary-Fine Decoupling for Action Generation in Robotic Imitationcited by 0
  38. 2026FocalPolicy: Frequency-Optimized Chunking and Locally Anchored Flow Matching for Coherent Visuomotor Policycited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization
  2. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
  3. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  4. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learning
  5. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  6. 2026COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control
  7. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models
  8. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model