Data to 14 Oct 2026

Online Bayesian Risk-Averse Reinforcement Learning

arXiv 2509.14077 ↗First posted 17 Sep 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In this paper, we study the Bayesian risk-averse formulation in reinforcement learning (RL). To address the epistemic uncertainty due to a lack of data, we adopt the Bayesian Risk Markov Decision Process (BRMDP) to account for the parameter uncertainty of the unknown underlying model. We derive the asymptotic normality that characterizes the difference between the Bayesian risk value function and the original value function under the true unknown distribution. The results indicate that the Bayesian risk-averse approach tends to pessimistically underestimate the original value function. This discrepancy increases with stronger risk aversion and decreases as more data become available. We then utilize this adaptive property in the setting of online RL as well as online contextual multi-arm bandits (CMAB), a special case of online RL. We provide two procedures using posterior sampling for both the general RL problem and the CMAB problem. We establish a sub-linear regret bound, with the regret defined as the conventional regret for both the RL and CMAB settings. Additionally, we establish a sub-linear regret bound for the CMAB setting with the regret defined as the Bayesian risk regret. Finally, we conduct numerical experiments to demonstrate the effectiveness of the proposed algorithm in addressing epistemic uncertainty and verifying the theoretical properties.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 38 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

21 of 38 references.

References · 21

2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2017A Tutorial on Thompson Samplingcited by 433
2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
2022Robust Reinforcement Learning using Offline Datacited by 96
2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
2019Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDPcited by 55
2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
2020Nearly Minimax Optimal Reinforcement Learning for Discounted MDPscited by 42
2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 36
2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36

9 more

2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
2020Learning Optimal Distributionally Robust Individualized Treatment Rulescited by 28
2020Distributionally Robust Batch Contextual Banditscited by 20
2022Online Policy Optimization for Robust MDPcited by 19
2023Wasserstein Distributionally Robust Policy Evaluation and Learning for Contextual Banditscited by 10
2021Bayesian Risk Markov Decision Processescited by 6
2023Bayesian Risk-Averse Q-Learning with Streaming Observationscited by 4
2023Approximate Bilevel Difference Convex Programming for Bayesian Risk Markov Decision Processescited by 2
2022Risk-averse Contextual Multi-armed Bandit Problem with Linear Payoffscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

2026Evolving Robustness--Exploration Trade-off in Online Reinforcement Learning via Quantile Bayesian Risk MDPscited by 0

Similar, not cited

Close in wording, no citation either way