# Risk-Averse Bayes-Adaptive Reinforcement Learning

arXiv: https://arxiv.org/abs/2102.05762 · HTML: https://mlsanity.com/paper/2102.05762/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 56 references that are in the map.
- First posted on arXiv 10 Feb 2021.

## Builds on (most cited first, 16 of 16)

- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [The Uncertainty Bellman Equation and Exploration](https://mlsanity.com/paper/1709.05380/) (2017)
- [Policy Gradient for Coherent Risk Measures](https://mlsanity.com/paper/1502.03919/) (2015)
- [Estimating Risk and Uncertainty in Deep Reinforcement Learning](https://mlsanity.com/paper/1905.09638/) (2019)
- [Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy](https://mlsanity.com/paper/1911.01546/) (2019)
- [Risk Averse Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1904.00511/) (2019)
- [Worst Cases Policy Gradients](https://mlsanity.com/paper/1911.03618/) (2019)
- [A Bayesian Approach to Robust Reinforcement Learning](https://mlsanity.com/paper/1905.08188/) (2019)
- [Policy Gradients for CVaR-Constrained MDPs](https://mlsanity.com/paper/1405.2690/) (2014)
- [Bayesian Optimized Monte Carlo Planning](https://mlsanity.com/paper/2010.03597/) (2020)
- [Robust and Adaptive Planning under Model Uncertainty](https://mlsanity.com/paper/1901.02577/) (2019)
- [Minimax Regret Optimisation for Robust Planning in Uncertain Markov Decision Processes](https://mlsanity.com/paper/2012.04626/) (2020)

## Built on by (most cited first, 29 of 29)

- [Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk](https://mlsanity.com/paper/2206.04436/) (2022)
- [One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2212.00124/) (2022)
- [Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief](https://mlsanity.com/paper/2210.06692/) (2022)
- [On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processes](https://mlsanity.com/paper/2304.12477/) (2023)
- [Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet](https://mlsanity.com/paper/2411.11451/) (2024)
- [Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments](https://mlsanity.com/paper/2504.19139/) (2025)
- [Decision-Making Under Uncertainty: Beyond Probabilities](https://mlsanity.com/paper/2303.05848/) (2023)
- [Two steps to risk sensitivity](https://mlsanity.com/paper/2111.06803/) (2021)
- [Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning](https://mlsanity.com/paper/2406.08002/) (2024)
- [Learning to Forecast Aleatoric and Epistemic Uncertainties over Long Horizon Trajectories](https://mlsanity.com/paper/2302.08669/) (2023)
- [Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation](https://mlsanity.com/paper/2504.20887/) (2025)
- [Planning for Risk-Aversion and Expected Value in MDPs](https://mlsanity.com/paper/2110.12746/) (2021)
- [Risk-Sensitive Bayesian Games for Multi-Agent Reinforcement Learning under Policy Uncertainty](https://mlsanity.com/paper/2203.10045/) (2022)
- [Merging Two Cultures: Deep and Statistical Learning](https://mlsanity.com/paper/2110.11561/) (2021)
- [Bayesian Risk-Averse Q-Learning with Streaming Observations](https://mlsanity.com/paper/2305.11300/) (2023)
- [Factorised Active Inference for Strategic Multi-Agent Interactions](https://mlsanity.com/paper/2411.07362/) (2024)
- [Learning to Be Cautious](https://mlsanity.com/paper/2110.15907/) (2021)
- [Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion](https://mlsanity.com/paper/2310.16546/) (2023)
- [Bayesian inference for data-efficient, explainable, and safe robotic motion planning: A review](https://mlsanity.com/paper/2307.08024/) (2023)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [On the Fundamental Limitations of Dual Static CVaR Decompositions in Markov Decision Processes](https://mlsanity.com/paper/2507.14005/) (2025)
- [Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism](https://mlsanity.com/paper/2512.04341/) (2025)
- [Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planning](https://mlsanity.com/paper/2304.00573/) (2023)
- [Adaptive Policy Learning to Additional Tasks](https://mlsanity.com/paper/2305.15193/) (2023)
- [Offline Bayesian Aleatoric and Epistemic Uncertainty Quantification and Posterior Value Optimisation in Finite-State MDPs](https://mlsanity.com/paper/2406.02456/) (2024)
- [Risk-averse policies for natural gas futures trading using distributional reinforcement learning](https://mlsanity.com/paper/2501.04421/) (2025)
- [Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief](https://mlsanity.com/paper/2606.00680/) (2026)
- [Efficient Bayes-Adaptive Reinforcement Learning with Temporal Logic Specifications](https://mlsanity.com/paper/2609.20954/) (2026)
- [Bridging Risk Approximation Gaps in Model Predictive Task Sampling via In-Context Modeling](https://mlsanity.com/paper/2610.14913/) (2026)

## Abstract (arXiv)

In this work, we address risk-averse Bayes-adaptive reinforcement learning. We pose the problem of optimising the conditional value at risk (CVaR) of the total return in Bayes-adaptive Markov decision processes (MDPs). We show that a policy optimising CVaR in this setting is risk-averse to both the parametric uncertainty due to the prior distribution over MDPs, and the internal uncertainty due to the inherent stochasticity of MDPs. We reformulate the problem as a two-player stochastic game and propose an approximate algorithm based on Monte Carlo tree search and Bayesian optimisation. Our experiments demonstrate that our approach significantly outperforms baseline approaches for this problem.