# Mitigating Planner Overfitting in Model-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/1812.01129 · HTML: https://mlsanity.com/paper/1812.01129/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2018.

## Built on by (most cited first, 9 of 9)

- [Toward Efficient Exploration by Large Language Model Agents](https://mlsanity.com/paper/2504.20997/) (2025)
- [Mind the Model, Not the Agent: The Primacy Bias in Model-based RL](https://mlsanity.com/paper/2310.15017/) (2023)
- [Planning to the Information Horizon of BAMDPs via Epistemic State Abstraction](https://mlsanity.com/paper/2210.16872/) (2022)
- [Dynamic Update-to-Data Ratio: Minimizing World Model Overfitting](https://mlsanity.com/paper/2303.10144/) (2023)
- [Imperfect World Models are Exploitable](https://mlsanity.com/paper/2605.15960/) (2026)
- [The Unintended Consequences of Discount Regularization: Improving Regularization in Certainty Equivalence Reinforcement Learning](https://mlsanity.com/paper/2306.11208/) (2023)
- [Comparison and Unification of Three Regularization Methods in Batch Reinforcement Learning](https://mlsanity.com/paper/2109.08134/) (2021)
- [A Bayesian Approach to Learning Bandit Structure in Markov Decision Processes](https://mlsanity.com/paper/2208.00250/) (2022)
- [POMRL: No-Regret Learning-to-Plan with Increasing Horizons](https://mlsanity.com/paper/2212.14530/) (2022)

## Abstract (arXiv)

An agent with an inaccurate model of its environment faces a difficult choice: it can ignore the errors in its model and act in the real world in whatever way it determines is optimal with respect to its model. Alternatively, it can take a more conservative stance and eschew its model in favor of optimizing its behavior solely via real-world interaction. This latter approach can be exceedingly slow to learn from experience, while the former can lead to "planner overfitting" - aspects of the agent's behavior are optimized to exploit errors in its model. This paper explores an intermediate position in which the planner seeks to avoid overfitting through a kind of regularization of the plans it considers. We present three different approaches that demonstrably mitigate planner overfitting in reinforcement-learning environments.