# A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning

arXiv: https://arxiv.org/abs/1806.07937 · HTML: https://mlsanity.com/paper/1806.07937/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2018.

## Built on by (most cited first, 30 of 87)

- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Replay-Guided Adversarial Environment Design](https://mlsanity.com/paper/2110.02439/) (2021)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [Decoupling Value and Policy for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2102.10330/) (2021)
- [XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAX](https://mlsanity.com/paper/2312.12044/) (2023)
- [Exploration via Elliptical Episodic Bonuses](https://mlsanity.com/paper/2210.05805/) (2022)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Adversarially Guided Actor-Critic](https://mlsanity.com/paper/2102.04376/) (2021)
- [SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies](https://mlsanity.com/paper/2106.09678/) (2021)
- [AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents](https://mlsanity.com/paper/2310.09971/) (2023)
- [Melting Pot 2.0](https://mlsanity.com/paper/2211.13746/) (2022)
- [Maximum Likelihood Reinforcement Learning](https://mlsanity.com/paper/2602.02710/) (2026)
- [Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning](https://mlsanity.com/paper/2209.09203/) (2022)
- [Continuous Coordination As a Realistic Scenario for Lifelong Learning](https://mlsanity.com/paper/2103.03216/) (2021)
- [On the Importance of Exploration for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2306.05483/) (2023)
- [Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments](https://mlsanity.com/paper/2101.08152/) (2021)
- [Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View](https://mlsanity.com/paper/2401.11237/) (2024)
- [Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward](https://mlsanity.com/paper/2512.16912/) (2025)
- [The Generalization Gap in Offline Reinforcement Learning](https://mlsanity.com/paper/2312.05742/) (2023)
- [Investigating the Properties of Neural Network Representations in Reinforcement Learning](https://mlsanity.com/paper/2203.15955/) (2022)
- [In-Context Reinforcement Learning for Variable Action Spaces](https://mlsanity.com/paper/2312.13327/) (2023)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [When Is Generalizable Reinforcement Learning Tractable?](https://mlsanity.com/paper/2101.00300/) (2021)
- [CLUTR: Curriculum Learning via Unsupervised Task Representation Learning](https://mlsanity.com/paper/2210.10243/) (2022)

## Abstract (arXiv)

The risks and perils of overfitting in machine learning are well known. However most of the treatment of this, including diagnostic tools and remedies, was developed for the supervised learning case. In this work, we aim to offer new perspectives on the characterization and prevention of overfitting in deep Reinforcement Learning (RL) methods, with a particular focus on continuous domains. We examine several aspects, such as how to define and diagnose overfitting in MDPs, and how to reduce risks by injecting sufficient training diversity. This work complements recent findings on the brittleness of deep RL methods and offers practical observations for RL researchers and practitioners.