# Bayesian Risk Markov Decision Processes

arXiv: https://arxiv.org/abs/2106.02558 · HTML: https://mlsanity.com/paper/2106.02558/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Jun 2021.

## Built on by (most cited first, 6 of 6)

- [Bayesian Risk-Averse Q-Learning with Streaming Observations](https://mlsanity.com/paper/2305.11300/) (2023)
- [Conformal Risk-Averse Decision Making with Action Conditional Guarantee](https://mlsanity.com/paper/2606.05551/) (2026)
- [Online Bayesian Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2509.14077/) (2025)
- [Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions](https://mlsanity.com/paper/2509.15509/) (2025)
- [Provably Invincible Adversarial Attacks on Reinforcement Learning Systems: A Rate-Distortion Information-Theoretic Approach](https://mlsanity.com/paper/2510.13792/) (2025)
- [Robust Successor Features](https://mlsanity.com/paper/2609.31016/) (2026)

## Abstract (arXiv)

We consider finite-horizon Markov Decision Processes where parameters, such as transition probabilities, are unknown and estimated from data. The popular distributionally robust approach to addressing the parameter uncertainty can sometimes be overly conservative. In this paper, we propose a new formulation, Bayesian risk Markov Decision Process (BR-MDP), to address parameter uncertainty in MDPs, where a risk functional is applied in nested form to the expected total cost with respect to the Bayesian posterior distribution of the unknown parameters. The proposed formulation provides more flexible risk attitutes towards parameter uncertainty and takes into account the availability of data in future times stages. To solve the proposed formulation with the conditional value-at-risk (CVaR) risk functional, we propose an efficient approximation algorithm by deriving an analytical approximation of the value function and utilizing the convexity of CVaR. We demonstrate the empirical performance of the BR-MDP formulation and proposed algorithms on a gambler's betting problem and an inventory control problem.