# Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2012.09156 · HTML: https://mlsanity.com/paper/2012.09156/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Dec 2020.

## Built on by (most cited first, 21 of 21)

- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting](https://mlsanity.com/paper/2306.01984/) (2023)
- [SafeDiffuser: Safe Planning with Diffusion Probabilistic Models](https://mlsanity.com/paper/2306.00148/) (2023)
- [Investigating Compounding Prediction Errors in Learned Dynamics Models](https://mlsanity.com/paper/2203.09637/) (2022)
- [The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2311.00168/) (2023)
- [World Models via Policy-Guided Trajectory Diffusion](https://mlsanity.com/paper/2312.08533/) (2023)
- [Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans](https://mlsanity.com/paper/2310.19427/) (2023)
- [Learning Dynamics Models for Model Predictive Agents](https://mlsanity.com/paper/2109.14311/) (2021)
- [Scalable Offline Model-Based RL with Action Chunks](https://mlsanity.com/paper/2512.08108/) (2025)
- [On Rollouts in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2501.16918/) (2025)
- [DMWM: Dual-Mind World Model with Long-Term Imagination](https://mlsanity.com/paper/2502.07591/) (2025)
- [A Unified View on Solving Objective Mismatch in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2310.06253/) (2023)
- [Distributional Successor Features Enable Zero-Shot Policy Optimization](https://mlsanity.com/paper/2403.06328/) (2024)
- [The Surprising Difficulty of Search in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2601.21306/) (2026)
- [Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Error](https://mlsanity.com/paper/2504.01766/) (2025)
- [WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Control](https://mlsanity.com/paper/2602.14351/) (2026)
- [Multi-timestep models for Model-based Reinforcement Learning](https://mlsanity.com/paper/2310.05672/) (2023)
- [Learning Long-Horizon Predictions for Quadrotor Dynamics](https://mlsanity.com/paper/2407.12964/) (2024)
- [Logarithmic Regret and Polynomial Scaling in Online Multi-step-ahead Prediction](https://mlsanity.com/paper/2511.12467/) (2025)
- [Reinforcement Learning in System Identification](https://mlsanity.com/paper/2212.07123/) (2022)
- [Recover Triggered States: Protect Model Against Backdoor Attack in Reinforcement Learning](https://mlsanity.com/paper/2304.00252/) (2023)

## Abstract (arXiv)

Accurately predicting the dynamics of robotic systems is crucial for model-based control and reinforcement learning. The most common way to estimate dynamics is by fitting a one-step ahead prediction model and using it to recursively propagate the predicted state distribution over long horizons. Unfortunately, this approach is known to compound even small prediction errors, making long-term predictions inaccurate. In this paper, we propose a new parametrization to supervised learning on state-action data to stably predict at longer horizons -- that we call a trajectory-based model. This trajectory-based model takes an initial state, a future time index, and control parameters as inputs, and directly predicts the state at the future time index. Experimental results in simulated and real-world robotic tasks show that trajectory-based models yield significantly more accurate long term predictions, improved sample efficiency, and the ability to predict task reward. With these improved prediction properties, we conclude with a demonstration of methods for using the trajectory-based model for control.