mlsanityData to 7 Oct 2026

The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning

arXiv 2402.12527 ↗First posted 19 Feb 2024cs.LG cs.AIcited by 8

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning aims to train agents from pre-collected datasets. However, this comes with the added challenge of estimating the value of behaviors not covered in the dataset. Model-based methods offer a potential solution by training an approximate dynamics model, which then allows collection of additional synthetic data via rollouts in this model. The prevailing theory treats this approach as online RL in an approximate dynamics model, and any remaining performance gap is therefore understood as being due to dynamics model errors. In this paper, we analyze this assumption and investigate how popular algorithms perform as the learned dynamics model is improved. In contrast to both intuition and theory, if the learned dynamics model is replaced by the true error-free dynamics, existing model-based methods completely fail. This reveals a key oversight: The theoretical foundations assume sampling of full horizon rollouts in the learned dynamics model; however, in practice, the number of model-rollout steps is aggressively reduced to prevent accumulating errors. We show that this truncation of rollouts results in a set of edge-of-reach states at which we are effectively ``bootstrapping from the void.'' This triggers pathological value overestimation and complete performance collapse. We term this the edge-of-reach problem. Based on this new insight, we fill important gaps in existing theory, and reveal how prior model-based methods are primarily addressing the edge-of-reach problem, rather than model-inaccuracy as claimed. Finally, we propose Reach-Aware Value Learning (RAVL), a simple and robust method that directly addresses the edge-of-reach problem and hence - unlike existing methods - does not fail as the dynamics model is improved. Code open-sourced at: github.com/anyasims/edge-of-reach.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 39 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

29 of 39 references.

Closest ancestors · 15

Same direction, newest first

  1. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  2. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  3. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
3 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  8. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  9. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  10. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  11. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  12. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
2 more
  1. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 41
  2. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 3.

8 papers.

  1. 2024Policy-Guided Diffusioncited by 40
  2. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  3. 2024Constrained Latent Action Policies for Model-Based Offline Reinforcement Learningcited by 3
  4. 2026Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLandercited by 2
  5. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
  6. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  7. 2026Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitionscited by 0
  8. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey on Offline Model-Based Reinforcement Learning
  2. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  3. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  4. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning
  5. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning
  6. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  7. 2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning
  8. 2022Model Generation with Provable Coverability for Offline Reinforcement Learning