mlsanityData to 7 Oct 2026

Investigating Compounding Prediction Errors in Learned Dynamics Models

arXiv 2203.09637 ↗First posted 17 Mar 2022cs.LGcited by 47

Not on the map. Why?

Abstract (arXiv)

Accurately predicting the consequences of agents' actions is a key prerequisite for planning in robotic control. Model-based reinforcement learning (MBRL) is one paradigm which relies on the iterative learning and prediction of state-action transitions to solve a task. Deep MBRL has become a popular candidate, using a neural network to learn a dynamics model that predicts with each pass from high-dimensional states to actions. These "one-step" predictions are known to become inaccurate over longer horizons of composed prediction - called the compounding error problem. Given the prevalence of the compounding error problem in MBRL and related fields of data-driven control, we set out to understand the properties of and conditions causing these long-horizon errors. In this paper, we explore the effects of subcomponents of a control problem on long term prediction error: including choosing a system, collecting data, and training a model. These detailed quantitative studies on simulated and real-world data show that the underlying dynamics of a system are the strongest factor determining the shape and magnitude of prediction error. Given a clearer understanding of compounding prediction error, researchers can implement new types of models beyond "one-step" that are more useful for control.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 21 of its 38 references that are in the map. First posted on arXiv 17 Mar 2022.

Builds on

21 of 38 references.

References · 21

  1. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
  4. 2020Lagrangian Neural Networkscited by 388
  5. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  6. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
  7. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  8. 2018Learning-based Model Predictive Control for Safe Explorationcited by 116
  9. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  10. 2020Objective Mismatch in Model-based Reinforcement Learningcited by 87
  11. 2021On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learningcited by 59
  12. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 52
9 more
  1. 2019Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learningcited by 40
  2. 2019Learning to Combat Compounding-Error in Model-Based Reinforcement Learningcited by 39
  3. 2015One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priorscited by 26
  4. 2020Structured Mechanical Models for Robot Learning and Controlcited by 21
  5. 2020Learning Accurate Long-term Dynamics for Model-based Reinforcement Learningcited by 21
  6. 2017Goal-Driven Dynamics Learning via Bayesian Optimizationcited by 19
  7. 2016Learning Quadrotor Dynamics Using Neural Network for Flight Controlcited by 15
  8. 2020Knowledge-Based Learning of Nonlinear Dynamics and Chaoscited by 14
  9. 2021Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?cited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 11; 2025: 16; 2026: 11.

47 papers.

Sort:

By area:

  1. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
  2. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  3. 2024Diffusion Model Predictive Controlcited by 35
  4. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  5. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  6. 2025PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulationcited by 22
  7. 2025Temporal Difference Flowscited by 20
  8. 2024A Distributional Analogue to the Successor Representationcited by 15
  9. 2025Improving Transformer World Models for Data-Efficient RLcited by 13
  10. 2022Goal-Space Planning with Subgoal Modelscited by 12
  11. 2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
  12. 2023TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matchingcited by 10
35 more
  1. 2023Self-Supervised Reinforcement Learning that Transfers using Random Featurescited by 10
  2. 2023A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learningcited by 9
  3. 2024WHALE: Towards Generalizable and Scalable World Models for Embodied Decision-makingcited by 9
  4. 2024Distributional Successor Features Enable Zero-Shot Policy Optimizationcited by 8
  5. 2025G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibrationcited by 8
  6. 2023What model does MuZero learn?cited by 7
  7. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  8. 2025Echoes of Discord: Forecasting Hater Reactions to Counterspeechcited by 5
  9. 2025Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Errorcited by 5
  10. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learningcited by 5
  11. 2025Time-Aware World Model for Adaptive Prediction and Controlcited by 4
  12. 2026Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safetycited by 4
  13. 2026Laplacian Representations for Decision-Time Planningcited by 4
  14. 2025First Order Model-Based RL through Decoupled Backpropagationcited by 3
  15. 2026A Definition and Roadmap for World Modelscited by 3
  16. 2023Multi-timestep models for Model-based Reinforcement Learningcited by 1
  17. 2024Deep Learning Based Simulators for the Phosphorus Removal Process Control in Wastewater Treatment via Deep Reinforcement Learning Algorithmscited by 1
  18. 2024A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learningcited by 1
  19. 2024Improved Long Short-Term Memory-based Wastewater Treatment Simulators for Deep Reinforcement Learningcited by 1
  20. 2024Learning Long-Horizon Predictions for Quadrotor Dynamicscited by 1
  21. 2024GenPlan: Generative Sequence Models as Adaptive Plannerscited by 1
  22. 2025Extendable Planning via Multiscale Diffusioncited by 1
  23. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  24. 2026Learning Sidewalk Autopilot from Multi-Scale Imitation with Corrective Behavior Expansioncited by 1
  25. 2026Latent Goal Prediction from Language for Model-Based Planningcited by 1
  26. 2022Reinforcement Learning in System Identificationcited by 0
  27. 2024A New View on Planning in Online Reinforcement Learningcited by 0
  28. 2025Learning with Expert Abstractions for Efficient Multi-Task Continuous Controlcited by 0
  29. 2025Physics-Informed Neural Controlled Differential Equations for Scalable Long Horizon Multi-Agent Motion Forecastingcited by 0
  30. 2025A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimizationcited by 0
  31. 2026Coupled Local and Global World Models for Efficient First Order RLcited by 0
  32. 2026Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimizationcited by 0
  33. 2026Reinforcement Learning: From Algorithms To Foundation Modelscited by 0
  34. 2026Dynamics Models for Offline Hyperparameter Selection in Real-World RLcited by 0
  35. 2026GPU-Accelerated Astrodynamics World Models for Spacecraft Rendezvous and Proximity Operationscited by 0