mlsanityData to 14 Oct 2026

Value-Distributional Model-Based Reinforcement Learning

arXiv 2308.06590 ↗First posted 12 Aug 2023cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Quantifying uncertainty about a policy's long-term performance is important to solve sequential decision-making tasks. We study the problem from a model-based Bayesian reinforcement learning perspective, where the goal is to learn the posterior distribution over value functions induced by parameter (epistemic) uncertainty of the Markov decision process. Previous work restricts the analysis to a few moments of the distribution over values or imposes a particular distribution shape, e.g., Gaussians. Inspired by distributional reinforcement learning, we introduce a Bellman operator whose fixed-point is the value distribution function. Based on our theory, we propose Epistemic Quantile-Regression (EQR), a model-based algorithm that learns a value distribution function. We combine EQR with soft actor-critic (SAC) for policy optimization with an arbitrary differentiable objective function of the learned value distribution. Evaluation across several continuous-control tasks shows performance benefits with respect to both model-based and model-free algorithms. The code is available at https://github.com/boschresearch/dist-mbrl.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 39 of its 62 references that are in the map. First posted on arXiv 12 Aug 2023.

Builds on

39 of 62 references.

References · 39

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  10. 2016Deep Exploration via Bootstrapped DQNcited by 527
  11. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
27 more
  1. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  2. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  3. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  4. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  5. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  6. 2017Deep Exploration via Randomized Value Functionscited by 156
  7. 2021Epistemic Neural Networkscited by 120
  8. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  9. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  10. 2017The Uncertainty Bellman Equation and Explorationcited by 82
  11. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  12. 2019Statistics and Samples in Distributional Reinforcement Learningcited by 63
  13. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  14. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  15. 2020Smooth Exploration for Robotic Reinforcement Learningcited by 54
  16. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  17. 2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 42
  18. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  19. 2019A Bayesian Approach to Robust Reinforcement Learningcited by 34
  20. 2023An Analysis of Quantile Temporal-Difference Learningcited by 31
  21. 2013Model-Based Bayesian Explorationcited by 21
  22. 2019Distributional reinforcement learning with linear function approximationcited by 16
  23. 2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
  24. 2023Model-Based Uncertainty in Value Functionscited by 15
  25. 2019Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansioncited by 13
  26. 2019Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimizationcited by 11
  27. 2023Distributional Model Equivalence for Risk-Sensitive Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observabilitycited by 5
  2. 2024Deep Exploration with PAC-Bayescited by 4
  3. 2025Overcoming Non-stationary Dynamics with Evidential Proximal Policy Optimizationcited by 2
  4. 2026SHAPO: Sharpness-Aware Policy Optimization for Safe Explorationcited by 2
  5. 2024Isoperimetry is All We Need: Langevin Posterior Sampling for RL with Sublinear Regretcited by 1