Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization

arXiv 2312.04386 ↗First posted 7 Dec 2023cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over Markov decision processes (MDPs). Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation (UBE), but the over-approximation may result in inefficient exploration. We propose a new UBE whose solution converges to the true posterior variance over values and leads to lower regret in tabular exploration problems. We identify challenges to apply the UBE theory beyond tabular problems and propose a suitable approximation. Based on this approximation, we introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC), that can be applied for either risk-seeking or risk-averse policy optimization with minimal changes. Experiments in both online and offline RL demonstrate improved performance compared to other uncertainty estimation methods.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 49 of its 71 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

49 of 71 references.

References · 49

2017Proximal Policy Optimization Algorithmscited by 15,215
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
2023Mastering Diverse Domains through World Modelscited by 1,253
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773

37 more

2017A Distributional Perspective on Reinforcement Learningcited by 765
2020MOPO: Model-based Offline Policy Optimizationcited by 636
2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
2016Deep Exploration via Bootstrapped DQNcited by 527
2020dm_control: Software and Tasks for Continuous Controlcited by 375
2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 302
2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
2017Deep Exploration via Randomized Value Functionscited by 156
2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
2019Better Exploration with Optimistic Actor-Criticcited by 88
2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
2017The Uncertainty Bellman Equation and Explorationcited by 82
2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
2020Smooth Exploration for Robotic Reinforcement Learningcited by 54
2021MBRL-Lib: A Modular Library for Model-based Reinforcement Learningcited by 42
2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
2020Making Sense of Reinforcement Learning and Probabilistic Inferencecited by 30
2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
2022From Dirichlet to Rubin: Optimistic Exploration in RL without Bonusescited by 22
2013Model-Based Bayesian Explorationcited by 21
2020Model-based Reinforcement Learning for Continuous Control with Posterior Samplingcited by 21
2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
2021Bayesian Bellman Operatorscited by 16
2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimizationcited by 16
2021SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learningcited by 15
2023Model-Based Uncertainty in Value Functionscited by 15
2019Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansioncited by 13
2019Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimizationcited by 11
2021On-Policy Model Errors in Reinforcement Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

2024Trust the Model Where It Trusts Itself -- Model-Based Actor-Critic with Uncertainty-Aware Rollout Adaptioncited by 18
2025On Rollouts in Model-Based Reinforcement Learningcited by 10
2024Isoperimetry is All We Need: Langevin Posterior Sampling for RL with Sublinear Regretcited by 1

Similar, not cited

Close in wording, no citation either way