# Self-Correcting Models for Model-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/1612.06018 · HTML: https://mlsanity.com/paper/1612.06018/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2016.

## Built on by (most cited first, 30 of 43)

- [Learning One Representation to Optimize All Rewards](https://mlsanity.com/paper/2103.07945/) (2021)
- [Value Gradient weighted Model-Based Reinforcement Learning](https://mlsanity.com/paper/2204.01464/) (2022)
- [Investigating the Properties of Neural Network Representations in Reinforcement Learning](https://mlsanity.com/paper/2203.15955/) (2022)
- [Goal-Space Planning with Subgoal Models](https://mlsanity.com/paper/2206.02902/) (2022)
- [Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse](https://mlsanity.com/paper/2305.18443/) (2023)
- [Locality Sensitive Sparse Encoding for Learning World Models Online](https://mlsanity.com/paper/2401.13034/) (2024)
- [High-dimensional separability for one- and few-shot learning](https://mlsanity.com/paper/2106.15416/) (2021)
- [Temporally Abstract Partial Models](https://mlsanity.com/paper/2108.03213/) (2021)
- [Harnessing Discrete Representations For Continual Reinforcement Learning](https://mlsanity.com/paper/2312.01203/) (2023)
- [On Rollouts in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2501.16918/) (2025)
- [A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2310.06253/) (2023)
- [A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning](https://mlsanity.com/paper/2404.09946/) (2024)
- [On-Policy Model Errors in Reinforcement Learning](https://mlsanity.com/paper/2110.07985/) (2021)
- [No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RL](https://mlsanity.com/paper/2205.08716/) (2022)
- [Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gap](https://mlsanity.com/paper/2507.00075/) (2025)
- [Laplacian Representations for Decision-Time Planning](https://mlsanity.com/paper/2602.05031/) (2026)
- [A Definition and Roadmap for World Models](https://mlsanity.com/paper/2607.06401/) (2026)
- [Learning the Reward Function for a Misspecified Model](https://mlsanity.com/paper/1801.09624/) (2018)
- [Monte Carlo Tree Search in the Presence of Transition Uncertainty](https://mlsanity.com/paper/2312.11348/) (2023)
- [WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Control](https://mlsanity.com/paper/2602.14351/) (2026)
- [Looped World Models](https://mlsanity.com/paper/2606.18208/) (2026)
- [The Intervention Gap in Latent World Models](https://mlsanity.com/paper/2608.29998/) (2026)
- [A Tractable Inference Perspective of Offline RL](https://mlsanity.com/paper/2311.00094/) (2023)
- [Maximum Entropy Model Correction in Reinforcement Learning](https://mlsanity.com/paper/2311.17855/) (2023)
- [Calibrated Value-Aware Model Learning with Probabilistic Environment Models](https://mlsanity.com/paper/2505.22772/) (2025)
- [λ-models: Effective Decision-Aware Reinforcement Learning with Latent Models](https://mlsanity.com/paper/2306.17366/) (2023)
- [Multi-timestep models for Model-based Reinforcement Learning](https://mlsanity.com/paper/2310.05672/) (2023)
- [A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learning](https://mlsanity.com/paper/2402.03146/) (2024)
- [Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning](https://mlsanity.com/paper/2405.03878/) (2024)
- [Bounding-Box Inference for Error-Aware Model-Based Reinforcement Learning](https://mlsanity.com/paper/2406.16006/) (2024)

## Abstract (arXiv)

When an agent cannot represent a perfectly accurate model of its environment's dynamics, model-based reinforcement learning (MBRL) can fail catastrophically. Planning involves composing the predictions of the model; when flawed predictions are composed, even minor errors can compound and render the model useless for planning. Hallucinated Replay (Talvitie 2014) trains the model to "correct" itself when it produces errors, substantially improving MBRL with flawed models. This paper theoretically analyzes this approach, illuminates settings in which it is likely to be effective or ineffective, and presents a novel error bound, showing that a model's ability to self-correct is more tightly related to MBRL performance than one-step prediction error. These results inspire an MBRL algorithm for deterministic MDPs with performance guarantees that are robust to model class limitations.