mlsanityData to 7 Oct 2026

Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning

arXiv 2405.17031 ↗First posted 27 May 2024cs.LGcited by 10

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based methods in reinforcement learning offer a promising approach to enhance data efficiency by facilitating policy exploration within a dynamics model. However, accurately predicting sequential steps in the dynamics model remains a challenge due to the bootstrapping prediction, which attributes the next state to the prediction of the current state. This leads to accumulated errors during model roll-out. In this paper, we propose the Any-step Dynamics Model (ADM) to mitigate the compounding error by reducing bootstrapping prediction to direct prediction. ADM allows for the use of variable-length plans as inputs for predicting future states without frequent bootstrapping. We design two algorithms, ADMPO-ON and ADMPO-OFF, which apply ADM in online and offline model-based frameworks, respectively. In the online setting, ADMPO-ON demonstrates improved sample efficiency compared to previous state-of-the-art methods. In the offline setting, ADMPO-OFF not only demonstrates superior performance compared to recent state-of-the-art offline approaches but also offers better quantification of model uncertainty using only a single ADM.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 53 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

31 of 53 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning9 Oct
  2. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization7 Oct
  3. 2022Offline Reinforcement Learning with Causal Structured World Models3 Jun
  4. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  5. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  10. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  6. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  7. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  8. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  9. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  10. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  11. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 52
  12. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 52
6 more
  1. 2020Bidirectional Model-based Policy Optimizationcited by 38
  2. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  3. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  4. 2017Path Integral Networks: End-to-End Differentiable Optimal Controlcited by 16
  5. 2022Adversarial Counterfactual Environment Model Learningcited by 12
  6. 2018Towards a Simple Approach to Multi-step Model-based Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 8.

10 papers.

  1. 2024WHALE: Towards Generalizable and Scalable World Models for Embodied Decision-makingcited by 9
  2. 2026Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Modelscited by 4
  3. 2026Speedup Patch: Learning a Plug-and-Play Policy to Accelerate Embodied Manipulationcited by 3
  4. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  5. 2026Robust Regularized Policy Iteration under Transition Uncertaintycited by 2
  6. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  7. 2026Offline Policy Optimization with Posterior Samplingcited by 1
  8. 2026Offline Reinforcement Learning with Universal Horizon Modelscited by 1
  9. 2026ReinVBC: A Model-based Reinforcement Learning Approach to Vehicle Braking Controllercited by 0
  10. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
  2. 2024A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learning
  3. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
  4. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  5. 2023A Survey on Offline Model-Based Reinforcement Learning
  6. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  7. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  8. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning