mlsanityData to 14 Oct 2026

Learning to Combat Compounding-Error in Model-Based Reinforcement Learning

arXiv 1912.11206 ↗First posted 24 Dec 2019cs.LG stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

Despite its potential to improve sample complexity versus model-free approaches, model-based reinforcement learning can fail catastrophically if the model is inaccurate. An algorithm should ideally be able to trust an imperfect model over a reasonably long planning horizon, and only rely on model-free updates when the model errors get infeasibly large. In this paper, we investigate techniques for choosing the planning horizon on a state-dependent basis, where a state's planning horizon is determined by the maximum cumulative model error around that state. We demonstrate that these state-dependent model errors can be learned with Temporal Difference methods, based on a novel approach of temporally decomposing the cumulative model errors. Experimental results show that the proposed method can successfully adapt the planning horizon to account for state-dependent model accuracy, significantly improving the efficiency of policy learning compared to model-based and model-free baselines.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 7; 2024: 10; 2025: 6; 2026: 9.

39 papers.

Sort:

By area:

  1. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  2. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  3. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 75
  4. 2025WorldGym: World Model as An Environment for Policy Evaluationcited by 64
  5. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  6. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  7. 2024Diffusion Model Predictive Controlcited by 36
  8. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  9. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 31
  10. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 21
  11. 2023Self-Supervised Reinforcement Learning that Transfers using Random Featurescited by 11
  12. 2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
27 more
  1. 2022Mingling Foresight with Imagination: Model-Based Cooperative Multi-Agent Reinforcement Learningcited by 6
  2. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 6
  3. 2024Learning to Play Atari in a World of Tokenscited by 6
  4. 2025First Order Model-Based RL through Decoupled Backpropagationcited by 5
  5. 2023Monte Carlo Tree Search in the Presence of Transition Uncertaintycited by 4
  6. 2022Revisiting Model-based Value Expansioncited by 3
  7. 2025Deep SPI: Safe Policy Improvement via World Modelscited by 3
  8. 2026WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Controlcited by 3
  9. 2026Looped World Modelscited by 3
  10. 2021Explaining Off-Policy Actor-Critic From A Bias-Variance Perspectivecited by 2
  11. 2022Backward Imitation and Forward Reinforcement Learning via Bi-directional Model Rolloutscited by 2
  12. 2024Denoising Diffusion Planner: Learning Complex Paths from Low-Quality Demonstrationscited by 2
  13. 2025Learning to Walk With Less: A Dyna-Style Approach to Quadrupedal Locomotioncited by 2
  14. 2026Extracting Search Trees from LLM Reasoning Traces Reveals Myopic Planningcited by 2
  15. 2026Latent Goal Prediction from Language for Model-Based Planningcited by 2
  16. 2023Reinforcement Learning under External Influence: Guarantees, Algorithms, and Sample Complexitycited by 1
  17. 2023Multi-timestep models for Model-based Reinforcement Learningcited by 1
  18. 2024Deep Learning Based Simulators for the Phosphorus Removal Process Control in Wastewater Treatment via Deep Reinforcement Learning Algorithmscited by 1
  19. 2024Learning Long-Horizon Predictions for Quadrotor Dynamicscited by 1
  20. 2026Coupled Local and Global World Models for Efficient First Order RLcited by 1
  21. 2026Advantage-Guided Diffusion for Model-Based Reinforcement Learningcited by 1
  22. 2026Dream-MPC: Gradient-Based Model Predictive Control with Latent Imaginationcited by 1
  23. 2022Reinforcement Learning in System Identificationcited by 0
  24. 2024Diminishing Return of Value Expansion Methodscited by 0
  25. 2025Heterogeneous Decision Making in Mixed Traffic: Uncertainty-aware Planning and Bounded Rationalitycited by 0
  26. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Criticcited by 0
  27. 2026Reinforcement Learning: From Algorithms To Foundation Modelscited by 0