mlsanityData to 14 Oct 2026

Quantifying Generalization in Reinforcement Learning

arXiv 1812.02341 ↗First posted 6 Dec 2018cs.LG stat.MLcited by 352

Not on the map. Why?

Cited by 92 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we investigate the problem of overfitting in deep reinforcement learning. Among the most common benchmarks in RL, it is customary to use the same environments for both training and testing. This practice offers relatively little insight into an agent's ability to generalize. We address this issue by using procedurally generated environments to construct distinct training and test sets. Most notably, we introduce a new environment called CoinRun, designed as a benchmark for generalization in RL. Using CoinRun, we find that agents overfit to surprisingly large training sets. We then show that deeper convolutional architectures improve generalization, as do methods traditionally found in supervised learning, including L2 regularization, dropout, data augmentation and batch normalization.

Key facts. Cited by 352 papers in the map (data through 14 Oct 2026). First posted on arXiv 6 Dec 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 2; 2020: 1; 2021: 65; 2022: 67; 2023: 52; 2024: 61; 2025: 62; 2026: 42.

352 papers.

Sort:

By area:

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2021Domain Generalization: A Surveycited by 764
  3. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  4. 2021Domain Generalization with MixStylecited by 566
  5. 2021Unsolved Problems in ML Safetycited by 292
  6. 2021MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learningcited by 290
  7. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  8. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  9. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  10. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  11. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  12. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
38 more
  1. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  2. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  3. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  4. 2023Learning to Act without Actionscited by 121
  5. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 115
  6. 2021Replay-Guided Adversarial Environment Designcited by 111
  7. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  8. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  9. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  10. 2023CelebV-Text: A Large-Scale Facial Text-Video Datasetcited by 104
  11. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
  12. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  13. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  14. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  15. 2022Make-A-Story: Visual Memory Conditioned Consistent Story Generationcited by 76
  16. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
  17. 2021Vision-Language Navigation with Random Environmental Mixupcited by 70
  18. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  19. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  20. 2021MixStyle Neural Networks for Domain Generalization and Adaptationcited by 64
  21. 2022Test-Time Adaptation via Self-Training with Nearest Neighbor Informationcited by 64
  22. 2023Scallop: A Language for Neurosymbolic Programmingcited by 63
  23. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 60
  24. 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
  25. 2022Exploration via Elliptical Episodic Bonusescited by 51
  26. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  27. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 45
  28. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 43
  29. 2021CompilerGym: Robust, Performant Compiler Optimization Environments for AI Researchcited by 41
  30. 2022Melting Pot 2.0cited by 40
  31. 2021How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Reviewcited by 39
  32. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 39
  33. 2021Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agentscited by 36
  34. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  35. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 34
  36. 2021Continuous Coordination As a Realistic Scenario for Lifelong Learningcited by 32
  37. 2022Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generationcited by 32
  38. 2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31

50 most cited of 352 All 352 (JSON)