# Investigating Compounding Prediction Errors in Learned Dynamics Models

arXiv: https://arxiv.org/abs/2203.09637 · HTML: https://mlsanity.com/paper/2203.09637/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 38 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Learning-based Model Predictive Control for Safe Exploration](https://mlsanity.com/paper/1803.08287/) (2018)
- [Model-Based Reinforcement Learning via Meta-Policy Optimization](https://mlsanity.com/paper/1809.05214/) (2018)
- [Objective Mismatch in Model-based Reinforcement Learning](https://mlsanity.com/paper/2002.04523/) (2020)
- [On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning](https://mlsanity.com/paper/2102.13651/) (2021)
- [Combating the Compounding-Error Problem with a Multi-step Model](https://mlsanity.com/paper/1905.13320/) (2019)
- [Low Level Control of a Quadrotor with Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/1901.03737/) (2019)
- [Learning to Combat Compounding-Error in Model-Based Reinforcement Learning](https://mlsanity.com/paper/1912.11206/) (2019)
- [One-Shot Learning of Manipulation Skills with Online Dynamics Adaptation and Neural Network Priors](https://mlsanity.com/paper/1509.06841/) (2015)
- [Structured Mechanical Models for Robot Learning and Control](https://mlsanity.com/paper/2004.10301/) (2020)
- [Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning](https://mlsanity.com/paper/2012.09156/) (2020)
- [Goal-Driven Dynamics Learning via Bayesian Optimization](https://mlsanity.com/paper/1703.09260/) (2017)
- [Learning Quadrotor Dynamics Using Neural Network for Flight Control](https://mlsanity.com/paper/1610.05863/) (2016)
- [Knowledge-Based Learning of Nonlinear Dynamics and Chaos](https://mlsanity.com/paper/2010.03415/) (2020)
- [Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose?](https://mlsanity.com/paper/2107.11587/) (2021)

## Built on by (most cited first, 30 of 47)

- [Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models](https://mlsanity.com/paper/2502.14819/) (2025)
- [The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2311.00168/) (2023)
- [Diffusion Model Predictive Control](https://mlsanity.com/paper/2410.05364/) (2024)
- [World Models via Policy-Guided Trajectory Diffusion](https://mlsanity.com/paper/2312.08533/) (2023)
- [Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.03570/) (2024)
- [Temporal Difference Flows](https://mlsanity.com/paper/2503.09817/) (2025)
- [PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation](https://mlsanity.com/paper/2504.16693/) (2025)
- [A Distributional Analogue to the Successor Representation](https://mlsanity.com/paper/2402.08530/) (2024)
- [Improving Transformer World Models for Data-Efficient RL](https://mlsanity.com/paper/2502.01591/) (2025)
- [Goal-Space Planning with Subgoal Models](https://mlsanity.com/paper/2206.02902/) (2022)
- [Self-Supervised Reinforcement Learning that Transfers using Random Features](https://mlsanity.com/paper/2305.17250/) (2023)
- [Implicit Search via Discrete Diffusion: A Study on Chess](https://mlsanity.com/paper/2502.19805/) (2025)
- [TOM: Learning Policy-Aware Models for Model-Based Reinforcement Learning via Transition Occupancy Matching](https://mlsanity.com/paper/2305.12663/) (2023)
- [A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2310.06253/) (2023)
- [WHALE: Towards Generalizable and Scalable World Models for Embodied Decision-making](https://mlsanity.com/paper/2411.05619/) (2024)
- [Intention-Conditioned Flow Occupancy Models](https://mlsanity.com/paper/2506.08902/) (2025)
- [Distributional Successor Features Enable Zero-Shot Policy Optimization](https://mlsanity.com/paper/2403.06328/) (2024)
- [G-Sim: Generative Simulations with Large Language Models and Gradient-Free Calibration](https://mlsanity.com/paper/2506.09272/) (2025)
- [What model does MuZero learn?](https://mlsanity.com/paper/2306.00840/) (2023)
- [Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Error](https://mlsanity.com/paper/2504.01766/) (2025)
- [The Surprising Difficulty of Search in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2601.21306/) (2026)
- [Echoes of Discord: Forecasting Hater Reactions to Counterspeech](https://mlsanity.com/paper/2501.16235/) (2025)
- [First Order Model-Based RL through Decoupled Backpropagation](https://mlsanity.com/paper/2509.00215/) (2025)
- [Laplacian Representations for Decision-Time Planning](https://mlsanity.com/paper/2602.05031/) (2026)
- [A Definition and Roadmap for World Models](https://mlsanity.com/paper/2607.06401/) (2026)
- [Time-Aware World Model for Adaptive Prediction and Control](https://mlsanity.com/paper/2506.08441/) (2025)
- [Beyond Simulations: What 20,000 Real Conversations Reveal About Mental Health AI Safety](https://mlsanity.com/paper/2601.17003/) (2026)
- [Extendable Planning via Multiscale Diffusion](https://mlsanity.com/paper/2503.20102/) (2025)
- [Latent Goal Prediction from Language for Model-Based Planning](https://mlsanity.com/paper/2606.20627/) (2026)
- [Multi-timestep models for Model-based Reinforcement Learning](https://mlsanity.com/paper/2310.05672/) (2023)

## Abstract (arXiv)

Accurately predicting the consequences of agents' actions is a key prerequisite for planning in robotic control. Model-based reinforcement learning (MBRL) is one paradigm which relies on the iterative learning and prediction of state-action transitions to solve a task. Deep MBRL has become a popular candidate, using a neural network to learn a dynamics model that predicts with each pass from high-dimensional states to actions. These "one-step" predictions are known to become inaccurate over longer horizons of composed prediction - called the compounding error problem. Given the prevalence of the compounding error problem in MBRL and related fields of data-driven control, we set out to understand the properties of and conditions causing these long-horizon errors. In this paper, we explore the effects of subcomponents of a control problem on long term prediction error: including choosing a system, collecting data, and training a model. These detailed quantitative studies on simulated and real-world data show that the underlying dynamics of a system are the strongest factor determining the shape and magnitude of prediction error. Given a clearer understanding of compounding prediction error, researchers can implement new types of models beyond "one-step" that are more useful for control.