Data to 7 Oct 2026

Combating the Compounding-Error Problem with a Multi-step Model

arXiv 1905.13320 ↗First posted 30 May 2019cs.LG cs.AI stat.MLcited by 52

Not on the map. Why?

Abstract (arXiv)

Model-based reinforcement learning is an appealing framework for creating agents that learn, plan, and act in sequential environments. Model-based algorithms typically involve learning a transition model that takes a state and an action and outputs the next state---a one-step model. This model can be composed with itself to enable predicting multiple steps into the future, but one-step prediction errors can get magnified, leading to unacceptable inaccuracy. This compounding-error problem plagues planning and undermines model-based reinforcement learning. In this paper, we address the compounding-error problem by introducing a multi-step model that directly outputs the outcome of executing a sequence of actions. Novel theoretical and empirical results indicate that the multi-step model is more conducive to efficient value-function estimation, and it yields better action selection compared to the one-step model. These results make a strong case for using multi-step models in the context of model-based reinforcement learning.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 8; 2023: 5; 2024: 11; 2025: 9; 2026: 12.

52 papers.

Sort:

By area:

2021Mastering Atari Games with Limited Datacited by 235
2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
2022A Survey on Model-based Reinforcement Learningcited by 65
2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
2021Mismatched No More: Joint Model-Policy Optimization for Model-Based RLcited by 35
2024Diffusion Model Predictive Controlcited by 35
2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
2022Learning continuous models for continuous physicscited by 28
2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
2021Sample Efficient Reinforcement Learning via Model-Ensemble Exploration and Exploitationcited by 17

38 more

2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
2025Scalable Offline Model-Based RL with Action Chunkscited by 12
2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learningcited by 10
2025On Rollouts in Model-Based Reinforcement Learningcited by 10
2024WHALE: Towards Generalizable and Scalable World Models for Embodied Decision-makingcited by 9
2023Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrationscited by 8
2021Robust Model-based Reinforcement Learning for Autonomous Greenhouse Controlcited by 6
2022Model-based Reinforcement Learning with Multi-step Plan Value Estimationcited by 6
2023How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimizationcited by 6
2024Benchmarking General-Purpose In-Context Learningcited by 6
2025Learning with Imperfect Models: When Multi-step Prediction Mitigates Compounding Errorcited by 5
2022Safe Reinforcement Learning using Data-Driven Predictive Controlcited by 4
2024Planning Transformer: Long-Horizon Offline Reinforcement Learning with Planning Tokenscited by 4
2025Adversarial Diffusion for Robust Reinforcement Learningcited by 4
2026WIMLE: Uncertainty-Aware World Models with IMLE for Sample-Efficient Continuous Controlcited by 3
2024Denoising Diffusion Planner: Learning Complex Paths from Low-Quality Demonstrationscited by 2
2025Bounding Distributional Shifts in World Modeling through Novelty Detectioncited by 2
2021Learning Generalizable Behavior via Visual Rewrite Rulescited by 1
2023Deep Generative Models for Decision-Making and Controlcited by 1
2023Multi-timestep models for Model-based Reinforcement Learningcited by 1
2024Deep Learning Based Simulators for the Phosphorus Removal Process Control in Wastewater Treatment via Deep Reinforcement Learning Algorithmscited by 1
2024Learning Long-Horizon Predictions for Quadrotor Dynamicscited by 1
2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
2026Joint Learning of Hierarchical Neural Options and Abstract World Modelcited by 1
2026Dream-MPC: Gradient-Based Model Predictive Control with Latent Imaginationcited by 1
2026Rethinking Code Review in the Age of AI: A Vision for Agentic Code Reviewcited by 1
2021MBDP: A Model-based Approach to Achieve both Robustness and Sample Efficiency via Double Dropout Planningcited by 0
2021ED2: Environment Dynamics Decomposition World Models for Continuous Controlcited by 0
2022Model Generation with Provable Coverability for Offline Reinforcement Learningcited by 0
2025Controllable Flow Matching for Online Reinforcement Learningcited by 0
2026Advantage-Guided Diffusion for Model-Based Reinforcement Learningcited by 0
2026Autoregressive Diffusion World Models for Off-Policy Evaluation of LLM Agentscited by 0
2026HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learningcited by 0
2026Reinforcement Learning: From Algorithms To Foundation Modelscited by 0
2026Beyond Myopic World Models: Long-Horizon End-to-End Training for Direct Future Predictioncited by 0
2026Towards Zero-Shot Task Transfer with Neurosymbolic World Modelscited by 0

50 most cited of 52 All 52 (JSON)