mlsanityData to 7 Oct 2026

Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation

arXiv 2107.00644 ↗First posted 1 Jul 2021cs.LG cs.CV cs.ROcited by 101

Not on the map. Why?

Abstract (arXiv)

While agents trained by Reinforcement Learning (RL) can solve increasingly challenging tasks directly from visual observations, generalizing learned skills to novel environments remains very challenging. Extensive use of data augmentation is a promising technique for improving generalization in RL, but it is often found to decrease sample efficiency and can even lead to divergence. In this paper, we investigate causes of instability when using data augmentation in common off-policy RL algorithms. We identify two problems, both rooted in high-variance Q-targets. Based on our findings, we propose a simple yet effective technique for stabilizing this class of algorithms under augmentation. We perform extensive empirical evaluation of image-based RL using both ConvNets and Vision Transformers (ViT) on a family of benchmarks based on DeepMind Control Suite, as well as in robotic manipulation tasks. Our method greatly improves stability and sample efficiency of ConvNets under augmentation, and achieves generalization results competitive with state-of-the-art methods for image-based RL in environments with unseen visuals. We further show that our method scales to RL with ViT-based architectures, and that data augmentation may be especially important in this setting.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 69 of its 86 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

69 of 86 references.

References · 69

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  10. 2015Continuous control with deep reinforcement learningcited by 3,279
  11. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
38 more
  1. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  5. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  7. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  8. 2019Contrastive Multiview Codingcited by 1,297
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  11. 2016Colorful Image Colorizationcited by 1,113
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  13. 2015Prioritized Experience Replaycited by 1,070
  14. 2018DeepMind Control Suitecited by 935
  15. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  16. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  17. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  18. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  19. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  20. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  21. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  22. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  23. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  24. 2020Reinforcement Learning with Augmented Datacited by 452
  25. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  26. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  27. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  28. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  29. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  30. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  31. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  32. 2019Self-Supervised Exploration via Disagreementcited by 263
  33. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  34. 2017Noisy Networks for Explorationcited by 262
  35. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  36. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  37. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  38. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 20; 2023: 29; 2024: 24; 2025: 17; 2026: 8.

101 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  2. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  3. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  4. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  5. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  6. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  7. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  8. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  9. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  10. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  11. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  12. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
38 more
  1. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
  2. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  3. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  4. 2023Finetuning Offline World Models in the Real Worldcited by 39
  5. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
  6. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  7. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
  8. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  9. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 32
  10. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  11. 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
  12. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  13. 2025GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulationcited by 26
  14. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  15. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  16. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  17. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  18. 2022Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learningcited by 21
  19. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
  20. 2022Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learningcited by 19
  21. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
  22. 2021Vision-Guided Quadrupedal Locomotion in the Wild with Multi-Modal Delay Randomizationcited by 18
  23. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  24. 2022Hierarchical Decision Transformercited by 16
  25. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  26. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  27. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  28. 2023Robust Representation Learning by Clustering with Bisimulation Metrics for Visual Reinforcement Learning with Distractionscited by 13
  29. 2023Normalization Enhances Generalization in Visual Reinforcement Learningcited by 13
  30. 2023Improving Generalization in Visual Reinforcement Learning via Conflict-aware Gradient Agreement Augmentationcited by 13
  31. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
  32. 2026stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluationcited by 13
  33. 2022Evaluating Vision Transformer Methods for Deep Reinforcement Learning from Pixelscited by 12
  34. 2023A Survey of Demonstration Learningcited by 12
  35. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12
  36. 2024A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learningcited by 12
  37. 2024Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulationcited by 11
  38. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10

50 most cited of 101 All 101 (JSON)