mlsanityData to 14 Oct 2026

Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism

arXiv 2512.04341 ↗First posted 4 Dec 2025cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Popular offline reinforcement learning (RL) methods rely on explicit conservatism, penalizing out-of-dataset actions or restricting rollout horizons. We question the universality of this principle and revisit a complementary Bayesian perspective for test-time adaptation. By modeling a posterior over world models and training a history-dependent agent to maximize expected return, the Bayesian approach directly addresses epistemic uncertainty without explicit conservatism. We first illustrate in a bandit setting that Bayesianism excels on low-quality datasets where conservatism fails. Scaling to realistic tasks, we find that long-horizon rollouts are essential to control value overestimation once conservatism is removed. We introduce design choices that enable learning from long-horizon rollouts while mitigating compounding model errors, yielding our algorithm, NEUBAY, grounded in the neutral Bayesian principle. On D4RL and NeoRL benchmarks, NEUBAY is competitive with leading conservative algorithms, achieving new state-of-the-art on 7 datasets with rollout horizons of several hundred steps. Finally, we characterize datasets by quality and coverage to identify when NEUBAY is preferable to conservative methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 103 of its 140 references that are in the map. First posted on arXiv 4 Dec 2025.

Builds on

103 of 140 references.

Closest ancestors · 43

Same direction, newest first

  1. 2025VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning16 Apr
  2. 2025Balancing optimism and pessimism in offline-to-online learning12 Feb
  3. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model27 Oct
  4. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning15 Oct
  5. 2024SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning23 Aug
  6. 2024Offline Model-Based Reinforcement Learning with Anti-Exploration20 Aug
  7. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning30 Jun
  8. 2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning27 May
  9. 2024The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning19 Feb
  10. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization11 Jan
  11. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning9 Oct
  12. 2023Model-based Offline Reinforcement Learning with Count-based Conservatism21 Jul
31 more
  1. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  2. 2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learning10 Apr
  3. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  4. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  5. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning8 Dec
  6. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning30 Nov
  7. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  8. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning12 Oct
  9. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization7 Oct
  10. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  11. 2022User-Interactive Offline Reinforcement Learning21 May
  12. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  13. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  14. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  15. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  16. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning31 Jan
  17. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  18. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  19. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  20. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  21. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  22. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model13 Apr
  23. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  24. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  25. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  26. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  27. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  28. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  29. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  30. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  31. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  10. 2023Mastering Diverse Domains through World Modelscited by 1,253
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
38 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  2. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  7. 2016Deep Exploration via Bootstrapped DQNcited by 527
  8. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  9. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  10. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  11. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  12. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  13. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  14. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  15. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  16. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  17. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
  18. 2021Equinox: neural networks in JAX via callable PyTrees and filtered transformationscited by 148
  19. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
  20. 2020Model-Based Offline Planningcited by 129
  21. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  22. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  23. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  24. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 97
  25. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
  26. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 63
  27. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 62
  28. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 54
  29. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  30. 2025A Survey of In-Context Reinforcement Learningcited by 39
  31. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  32. 2020Trust the Model When It Is Confident: Masked Model-based Actor-Criticcited by 36
  33. 2024Diffusion Model Predictive Controlcited by 36
  34. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 34
  35. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 31
  36. 2021Risk-Averse Bayes-Adaptive Reinforcement Learningcited by 29
  37. 2021Recurrent Off-policy Baselines for Memory-based Continuous Controlcited by 21
  38. 2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 18

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  2. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  3. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  4. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
  5. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
  6. 2025Fully Offline Reinforcement Learning
  7. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  8. 2023Selective Uncertainty Propagation in Offline RL