# Combating the Compounding-Error Problem with a Multi-step Model

arXiv: https://arxiv.org/abs/1905.13320 · HTML: https://mlsanity.com/paper/1905.13320/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2019.

## Built on by (most cited first, 30 of 54)

- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences](https://mlsanity.com/paper/2401.10529/) (2024)
- [Safe Reinforcement Learning by Imagining the Near Future](https://mlsanity.com/paper/2202.07789/) (2022)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [Investigating Compounding Prediction Errors in Learned Dynamics Models](https://mlsanity.com/paper/2203.09637/) (2022)
- [Mismatched No More: Joint Model-Policy Optimization for Model-Based RL](https://mlsanity.com/paper/2110.02758/) (2021)
- [Diffusion Model Predictive Control](https://mlsanity.com/paper/2410.05364/) (2024)
- [Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.03570/) (2024)
- [Learning continuous models for continuous physics](https://mlsanity.com/paper/2202.08494/) (2022)
- [Learning Latent Dynamic Robust Representations for World Models](https://mlsanity.com/paper/2405.06263/) (2024)
- [Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function](https://mlsanity.com/paper/2302.01244/) (2023)
- [Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitation](https://mlsanity.com/paper/2107.01825/) (2021)
- [Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy](https://mlsanity.com/paper/2207.12141/) (2022)
- [Scalable Offline Model-Based RL with Action Chunks](https://mlsanity.com/paper/2512.08108/) (2025)
- [Implicit Search via Discrete Diffusion: A Study on Chess](https://mlsanity.com/paper/2502.19805/) (2025)
- [Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning](https://mlsanity.com/paper/2405.17031/) (2024)
- [On Rollouts in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2501.16918/) (2025)
- [WHALE: Towards Generalizable and Scalable World Models for Embodied Decision-making](https://mlsanity.com/paper/2411.05619/) (2024)
- [Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrations](https://mlsanity.com/paper/2312.11442/) (2023)
- [Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control](https://mlsanity.com/paper/2108.11645/) (2021)
- [Model-based Reinforcement Learning with Multi-step Plan Value Estimation](https://mlsanity.com/paper/2209.05530/) (2022)
- [How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimization](https://mlsanity.com/paper/2309.12671/) (2023)
- [Benchmarking General-Purpose In-Context Learning](https://mlsanity.com/paper/2405.17234/) (2024)
- [Safe Reinforcement Learning using Data-Driven Predictive Control](https://mlsanity.com/paper/2211.11027/) (2022)
- [Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Error](https://mlsanity.com/paper/2504.01766/) (2025)
- [Planning Transformer: Long-Horizon Offline Reinforcement Learning with Planning Tokens](https://mlsanity.com/paper/2409.09513/) (2024)
- [Adversarial Diffusion for Robust Reinforcement Learning](https://mlsanity.com/paper/2509.23846/) (2025)
- [WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Control](https://mlsanity.com/paper/2602.14351/) (2026)
- [Denoising Diffusion Planner: Learning Complex Paths from Low-Quality Demonstrations](https://mlsanity.com/paper/2410.21497/) (2024)
- [Bounding Distributional Shifts in World Modeling through Novelty Detection](https://mlsanity.com/paper/2508.06096/) (2025)

## Abstract (arXiv)

Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a state and an action and outputs the next state---a one-step model. This model can be composed with itself to enable predicting multiple steps into the future, but one-step prediction errors can get magnified, leading to unacceptable inaccuracy. This compounding-error problem plagues planning and undermines model-based reinforcement learning. In this paper, we address the compounding-error problem by introducing a multi-step model that directly outputs the outcome of executing a sequence of actions. Novel theoretical and empirical results indicate that the multi-step model is more conducive to efficient value-function estimation, and it yields better action selection compared to the one-step model. These results make a strong case for using multi-step models in the context of model-based reinforcement learning.