mlsanityData to 7 Oct 2026

Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updates

arXiv 2310.17786 ↗First posted 26 Oct 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Recently, data augmentation (DA) has emerged as a method for leveraging domain knowledge to inexpensively generate additional data in reinforcement learning (RL) tasks, often yielding substantial improvements in data efficiency. While prior work has demonstrated the utility of incorporating augmented data directly into model-free RL updates, it is not well-understood when a particular DA strategy will improve data efficiency. In this paper, we seek to identify general aspects of DA responsible for observed learning improvements. Our study focuses on sparse-reward tasks with dynamics-invariant data augmentation functions, serving as an initial step towards a more general understanding of DA and its integration into RL training. Experimentally, we isolate three relevant aspects of DA: state-action coverage, reward density, and the number of augmented transitions generated per update (the augmented replay ratio). From our experiments, we draw two conclusions: (1) increasing state-action coverage often has a much greater impact on data efficiency than increasing reward density, and (2) decreasing the augmented replay ratio substantially improves data efficiency. In fact, certain tasks in our empirical study are solvable only when the replay ratio is sufficiently low.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

40 of 59 references.

References · 40

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  8. 2020Reinforcement Learning with Augmented Datacited by 452
  9. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  12. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
28 more
  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  2. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  3. 2020Revisiting Fundamentals of Experience Replaycited by 171
  4. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  5. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  6. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  7. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 133
  8. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  9. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  10. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  11. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  12. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  13. 2021Residual Pathway Priors for Soft Equivariance Constraintscited by 74
  14. 2021Equivariant Q Learning in Spatial Action Spacescited by 61
  15. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 56
  16. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  17. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  18. 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
  19. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  20. 2021Efficient Differentiable Simulation of Articulated Bodiescited by 38
  21. 2017Hindsight policy gradientscited by 36
  22. 2022The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetrycited by 32
  23. 2021Multi-Agent MDP Homomorphic Networkscited by 24
  24. 2019Competitive Experience Replaycited by 23
  25. 2022Data Augmentation for Manipulationcited by 16
  26. 2020Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraftcited by 15
  27. 2018ARCHER: Aggressive Rewards to Counter bias in Hindsight Experience Replaycited by 9
  28. 2017Run, skeleton, run: skeletal model in a physics-based simulationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 1; 2026: 1.

6 papers.

  1. 2024Dream to Manipulate: Compositional World Models Empowering Robot Imitation Learning with Imaginationcited by 45
  2. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10
  3. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  4. 2024Reinforcement Learning with Euclidean Data Augmentation for State-Based Continuous Controlcited by 2
  5. 2025Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learningcited by 0
  6. 2026Reflex: Reinforcement Learning with Reflection Symmetry Exploitation in State-Based Continuous Controlcited by 0