mlsanityData to 7 Oct 2026

A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learning

arXiv 2405.17416 ↗First posted 27 May 2024cs.LG cs.CV cs.ROcited by 12

Not on the map. Why?

Abstract (arXiv)

Q-learning algorithms are appealing for real-world applications due to their data-efficiency, but they are very prone to overfitting and training instabilities when trained from visual observations. Prior work, namely SVEA, finds that selective application of data augmentation can improve the visual generalization of RL agents without destabilizing training. We revisit its recipe for data augmentation, and find an assumption that limits its effectiveness to augmentations of a photometric nature. Addressing these limitations, we propose a generalized recipe, SADA, that works with wider varieties of augmentations. We benchmark its effectiveness on DMC-GB2 - our proposed extension of the popular DMControl Generalization Benchmark - as well as tasks from Meta-World and the Distracting Control Suite, and find that our method, SADA, greatly improves training stability and generalization of RL agents across a diverse set of augmentations. For visualizations, code and benchmark: see https://aalmuzairee.github.io/SADA/

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 40 of its 45 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

40 of 45 references.

References · 40

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  8. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  9. 2019Contrastive Multiview Codingcited by 1,297
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  12. 2018DeepMind Control Suitecited by 935
28 more
  1. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  4. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  5. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  6. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  7. 2020Reinforcement Learning with Augmented Datacited by 452
  8. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  9. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  10. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  11. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  12. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  13. 2022Masked World Models for Visual Controlcited by 169
  14. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  15. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  16. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  17. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  18. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  19. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  20. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  21. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  22. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 42
  23. 2021Unsupervised Visual Attention and Invariance for Reinforcement Learningcited by 30
  24. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  25. 2022Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learningcited by 21
  26. 2021Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentationcited by 15
  27. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  28. 2023Generalizable Visual Reinforcement Learning with Segment Anything Modelcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 3.

12 papers.

  1. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  2. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  3. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  4. 2024Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulationcited by 11
  5. 2024Zero-Shot Generalization of Vision-Based RL Without Data Augmentationcited by 3
  6. 2025Zero-Shot Visual Generalization in Robot Manipulationcited by 3
  7. 2025Merging and Disentangling Views in Visual Reinforcement Learning for Robotic Manipulationcited by 2
  8. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  9. 2025SegDAC: Visual Generalization in Reinforcement Learning via Dynamic Object Tokenscited by 0
  10. 2025Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Datacited by 0
  11. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0
  12. 2026StarWM: Self-Supervised Trained Attention Routing for Robust World Modelscited by 0