mlsanityData to 14 Oct 2026

Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence

arXiv 2402.02701 ↗First posted 5 Feb 2024cs.LG cs.AI stat.MLcited by 5

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Recently, there are many efforts attempting to learn useful policies for continuous control in visual reinforcement learning (RL). In this scenario, it is important to learn a generalizable policy, as the testing environment may differ from the training environment, e.g., there exist distractors during deployment. Many practical algorithms are proposed to handle this problem. However, to the best of our knowledge, none of them provide a theoretical understanding of what affects the generalization gap and why their proposed methods work. In this paper, we bridge this issue by theoretically answering the key factors that contribute to the generalization gap when the testing environment has distractors. Our theories indicate that minimizing the representation distance between training and testing environments, which aligns with human intuition, is the most critical for the benefit of reducing the generalization gap. Our theoretical results are supported by the empirical evidence in the DMControl Generalization Benchmark (DMC-GB).

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 91 of its 116 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

91 of 116 references.

References · 91

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2023Segment Anythingcited by 8,234
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  9. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  10. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  11. 2021Domain Generalization: A Surveycited by 764
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
38 more
  1. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  2. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  3. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  4. 2020Reinforcement Learning with Augmented Datacited by 468
  5. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  6. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  7. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  8. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  9. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  10. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  11. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  12. 2017Unified Deep Supervised Domain Adaptation and Generalizationcited by 323
  13. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  14. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 293
  15. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  16. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  17. 2019Revisiting Self-Supervised Visual Representation Learningcited by 251
  18. 2019Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Datacited by 246
  19. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 238
  20. 2021Mastering Atari Games with Limited Datacited by 237
  21. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  22. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  23. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  24. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 201
  25. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 189
  26. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  27. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  28. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  29. 2018A Tour of Reinforcement Learning: The View from Continuous Controlcited by 151
  30. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  31. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  32. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  33. 2018Recent Advances in Autoencoder-Based Representation Learningcited by 133
  34. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  35. 2019Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Modelscited by 117
  36. 2020Naive Exploration is Optimal for Online LQRcited by 116
  37. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  38. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 104

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 41
  2. 2024Advancing DRL Agents in Commercial Fighting Games: Training, Integration, and Agent-Human Alignmentcited by 8
  3. 2025AbideGym: Turning Static RL Worlds into Adaptive Challengescited by 1
  4. 2026Toward Active Object Detection for UAVs in the Wild: A Large-Scale Dataset, Benchmark and Methodcited by 1
  5. 2025Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Focus On What Matters: Separated Models For Visual-Based RL Generalization
  2. 2026Why Does RL Generalize Better Than SFT? A Data-Centric Perspective on VLM Post-Training
  3. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation
  4. 2026VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning
  5. 2023The Generalization Gap in Offline Reinforcement Learning
  6. 2025Good Actions Succeed, Bad Actions Generalize: A Case Study on Why RL Generalizes Better
  7. 2024Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learning
  8. 2025Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning