mlsanityData to 7 Oct 2026

Reflect-then-Plan: Offline Model-Based Planning through a Doubly Bayesian Lens

arXiv 2506.06261 ↗First posted 6 Jun 2025cs.AI cs.LGcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) is crucial when online exploration is costly or unsafe but often struggles with high epistemic uncertainty due to limited data. Existing methods rely on fixed conservative policies, restricting adaptivity and generalization. To address this, we propose Reflect-then-Plan (RefPlan), a novel doubly Bayesian offline model-based (MB) planning approach. RefPlan unifies uncertainty modeling and MB planning by recasting planning as Bayesian posterior estimation. At deployment, it updates a belief over environment dynamics using real-time observations, incorporating uncertainty into MB planning via marginalization. Empirical results on standard benchmarks show that RefPlan significantly improves the performance of conservative offline RL policies. In particular, RefPlan maintains robust performance under high epistemic uncertainty and limited data, while demonstrating resilience to changing environment dynamics, improving the flexibility, generalizability, and robustness of offline-learned policies.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 40 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

27 of 40 references.

Closest ancestors · 11

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 16

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  6. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  7. 2018Maximum a Posteriori Policy Optimisationcited by 318
  8. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  9. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  10. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  11. 2020Model-Based Offline Planningcited by 126
  12. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
4 more
  1. 2018Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Controlcited by 104
  2. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  3. 2020Learning Off-Policy with Online Planningcited by 46
  4. 2019Variational Inference MPC for Bayesian Model-based Reinforcement Learningcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  2. 2025Align While Search: Belief-Guided Exploratory Inference for World-Grounded Embodied Agentscited by 0
  3. 2026VariBASed: Variational Bayes-Adaptive Sequential Monte-Carlo Planning for Deep Reinforcement Learningcited by 0
  4. 2026UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
  2. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  3. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
  4. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  5. 2026Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds
  6. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
  7. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
  8. 2020MOPO: Model-based Offline Policy Optimization