mlsanityData to 14 Oct 2026

A Study on Overfitting in Deep Reinforcement Learning

arXiv 1804.06893 ↗First posted 18 Apr 2018cs.LG stat.MLcited by 174

Not on the map. Why?

Cited by 48 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent years have witnessed significant progresses in deep Reinforcement Learning (RL). Empowered with large scale neural networks, carefully designed architectures, novel training algorithms and massively parallel computing devices, researchers are able to attack many challenging RL problems. However, in machine learning, more training power comes with a potential risk of more overfitting. As deep RL techniques are being applied to critical problems such as healthcare and finance, it is important to understand the generalization behaviors of the trained agents. In this paper, we conduct a systematic study of standard RL agents and find that they could overfit in various ways. Moreover, overfitting could happen "robustly": commonly used techniques in RL that add stochasticity do not necessarily prevent or detect overfitting. In particular, the same agents and learning algorithms could have drastically different test performance, even when all of them achieve optimal rewards during training. The observations call for more principled and careful evaluation protocols in RL. We conclude with a general discussion on overfitting in RL and a study of the generalization behaviors from the perspective of inductive bias.

Key facts. Cited by 174 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 34; 2022: 35; 2023: 32; 2024: 27; 2025: 30; 2026: 14.

174 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  3. 2021Domain Generalization with MixStylecited by 566
  4. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  5. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  6. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  7. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  8. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  9. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  10. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  11. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  12. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
38 more
  1. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  2. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  3. 2021A Survey on Interpretable Reinforcement Learningcited by 73
  4. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  5. 2021MixStyle Neural Networks for Domain Generalization and Adaptationcited by 64
  6. 2025Reinforcement Learning from Human Feedbackcited by 58
  7. 2022Exploration via Elliptical Episodic Bonusescited by 51
  8. 2021Adversarially Guided Actor-Criticcited by 47
  9. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  10. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 38
  11. 2026Maximum Likelihood Reinforcement Learningcited by 36
  12. 2022Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learningcited by 34
  13. 2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
  14. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 31
  15. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  16. 2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 30
  17. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  18. 2023The Generalization Gap in Offline Reinforcement Learningcited by 26
  19. 2023Causal Reinforcement Learning: A Surveycited by 23
  20. 2024MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learningcited by 23
  21. 2021Quantifying the effects of environment and population diversity in multi-agent reinforcement learningcited by 22
  22. 2022Don't Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learningcited by 21
  23. 2023Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programscited by 20
  24. 2023A Study of Global and Episodic Bonuses for Exploration in Contextual MDPscited by 20
  25. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  26. 2021Domain Adversarial Reinforcement Learningcited by 19
  27. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  28. 2022Emergent Bartering Behaviour in Multi-Agent Reinforcement Learningcited by 19
  29. 2022Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learningcited by 19
  30. 2021TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RLcited by 18
  31. 2022Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worldscited by 17
  32. 2025Horizon Generalization in Reinforcement Learningcited by 17
  33. 2021Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentationcited by 15
  34. 2024General-purpose foundation models for increased autonomy in robot-assisted surgerycited by 15
  35. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 14
  36. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  37. 2025Touch begins where vision ends: Generalizable policies for contact-rich manipulationcited by 14
  38. 2022Zipfian environments for Reinforcement Learningcited by 13

50 most cited of 174 All 174 (JSON)