mlsanityData to 7 Oct 2026

Self-Correcting Models for Model-Based Reinforcement Learning

arXiv 1612.06018 ↗First posted 19 Dec 2016cs.LG cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

When an agent cannot represent a perfectly accurate model of its environment's dynamics, model-based reinforcement learning (MBRL) can fail catastrophically. Planning involves composing the predictions of the model; when flawed predictions are composed, even minor errors can compound and render the model useless for planning. Hallucinated Replay (Talvitie 2014) trains the model to "correct" itself when it produces errors, substantially improving MBRL with flawed models. This paper theoretically analyzes this approach, illuminates settings in which it is likely to be effective or ineffective, and presents a novel error bound, showing that a model's ability to self-correct is more tightly related to MBRL performance than one-step prediction error. These results inspire an MBRL algorithm for deterministic MDPs with performance guarantees that are robust to model class limitations.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Dec 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 5; 2022: 5; 2023: 8; 2024: 8; 2025: 6; 2026: 7.

40 papers.

Sort:

By area:

  1. 2021Learning One Representation to Optimize All Rewardscited by 110
  2. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  3. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  4. 2022Goal-Space Planning with Subgoal Modelscited by 12
  5. 2023Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reusecited by 12
  6. 2024Locality Sensitive Sparse Encoding for Learning World Models Onlinecited by 12
  7. 2021High-dimensional separability for one- and few-shot learningcited by 11
  8. 2021Temporally Abstract Partial Modelscited by 11
  9. 2023Harnessing Discrete Representations For Continual Reinforcement Learningcited by 10
  10. 2025On Rollouts in Model-Based Reinforcement Learningcited by 10
  11. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  12. 2024A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learningcited by 7
28 more
  1. 2021On-Policy Model Errors in Reinforcement Learningcited by 6
  2. 2022No More Pesky Hyperparameters: Offline Hyperparameter Tuning for RLcited by 6
  3. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  4. 2018Learning the Reward Function for a Misspecified Modelcited by 4
  5. 2026Laplacian Representations for Decision-Time Planningcited by 4
  6. 2023Monte Carlo Tree Search in the Presence of Transition Uncertaintycited by 3
  7. 2026WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Controlcited by 3
  8. 2026A Definition and Roadmap for World Modelscited by 3
  9. 2023A Tractable Inference Perspective of Offline RLcited by 2
  10. 2023Maximum Entropy Model Correction in Reinforcement Learningcited by 2
  11. 2025Calibrated Value-Aware Model Learning with Probabilistic Environment Modelscited by 2
  12. 2026Looped World Modelscited by 2
  13. 2026The Intervention Gap in Latent World Modelscited by 2
  14. 2023λ-models: Effective Decision-Aware Reinforcement Learning with Latent Modelscited by 1
  15. 2023Multi-timestep models for Model-based Reinforcement Learningcited by 1
  16. 2024A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learningcited by 1
  17. 2024Sequence Compression Speeds Up Credit Assignment in Reinforcement Learningcited by 1
  18. 2024Bounding-Box Inference for Error-Aware Model-Based Reinforcement Learningcited by 1
  19. 2024Why long model-based rollouts are no reason for bad Q-value estimatescited by 1
  20. 2024Rejecting Hallucinated State Targets during Planningcited by 1
  21. 2025Situationally-Aware Dynamics Learningcited by 1
  22. 2021Planning with Expectation Models for Controlcited by 0
  23. 2022Reward-Predictive Clusteringcited by 0
  24. 2024Meta-Gradient Search Control: A Method for Improving the Efficiency of Dyna-style Planningcited by 0
  25. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
  26. 2025Planning as Descent: Goal-Conditioned Latent Trajectory Synthesis in Learned Energy Landscapescited by 0
  27. 2026Beyond Myopic World Models: Long-Horizon End-to-End Training for Direct Future Predictioncited by 0
  28. 2026Dynamics Models for Offline Hyperparameter Selection in Real-World RLcited by 0