mlsanityData to 7 Oct 2026

Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning

arXiv 2406.04088 ↗First posted 6 Jun 2024cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Current approaches to model-based offline reinforcement learning often incorporate uncertainty-based reward penalization to address the distributional shift problem. These approaches, commonly known as pessimistic value iteration, use Monte Carlo sampling to estimate the Bellman target to perform temporal difference-based policy evaluation. We find out that the randomness caused by this sampling step significantly delays convergence. We present a theoretical result demonstrating the strong dependency of suboptimality on the number of Monte Carlo samples taken per Bellman target calculation. Our main contribution is a deterministic approximation to the Bellman target that uses progressive moment matching, a method developed originally for deterministic variational inference. The resulting algorithm, which we call Moment Matching Offline Model-Based Policy Optimization (MOMBO), propagates the uncertainty of the next state through a nonlinear Q-network in a deterministic fashion by approximating the distributions of hidden layer activations by a normal distribution. We show that it is possible to provide tighter guarantees for the suboptimality of MOMBO than the existing Monte Carlo sampling approaches. We also observe MOMBO to converge faster than these approaches in a large set of benchmark tasks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 65 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

44 of 65 references.

Closest ancestors · 21

Same direction, newest first

  1. 2024Simple Ingredients for Offline Reinforcement Learning19 Mar
  2. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  3. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization7 Oct
  4. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  5. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  6. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  7. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning8 Oct
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
9 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2022Transformers are Sample-Efficient World Modelscited by 317
  12. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
11 more
  1. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  2. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  3. 2018Lightweight Probabilistic Deep Networkscited by 77
  4. 2019Learning When-to-Treat Policiescited by 49
  5. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  6. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  7. 2019Deep Active Learning with Adaptive Acquisitioncited by 17
  8. 2023Model-Based Uncertainty in Value Functionscited by 15
  9. 2024An Analytic Solution to Covariance Propagation in Neural Networkscited by 11
  10. 2023On the 1-Wasserstein Distance between Location-Scale Distributions and the Effect of Differential Privacycited by 7
  11. 2020A Deterministic Approximation to Neural SDEscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Assumed Density Filtering and Smoothing with Neural Network Surrogate Modelscited by 1
  2. 2026Analytic Planning under Uncertainty with Moment Closurecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Selective Uncertainty Propagation in Offline RL
  2. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  3. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism
  4. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning
  5. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  6. 2021DROMO: Distributionally Robust Offline Model-based Policy Optimization
  7. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
  8. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach