mlsanityData to 14 Oct 2026

Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions

arXiv 2509.15509 ↗First posted 19 Sep 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Motivated by many application problems, we consider Markov decision processes (MDPs) with a general loss function and unknown parameters. To mitigate the epistemic uncertainty associated with unknown parameters, we take a Bayesian approach to estimate the parameters from data and impose a coherent risk functional (with respect to the Bayesian posterior distribution) on the loss. Since this formulation usually does not satisfy the interchangeability principle, it does not admit Bellman equations and cannot be solved by approaches based on dynamic programming. Therefore, We propose a policy gradient optimization method, leveraging the dual representation of coherent risk measures and extending the envelope theorem to continuous cases. We then show the stationary analysis of the algorithm with a convergence rate of O(T-1/2+r-1/2), where T is the number of policy gradient iterations and r is the sample size of the gradient estimator. We further extend our algorithm to an episodic setting, and establish the global convergence of the extended algorithm and provide bounds on the number of iterations needed to achieve an error bound O(ε) in each episode.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 35 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

11 of 35 references.

References · 11

  1. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  2. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  3. 2014Algorithms for CVaR Optimization in MDPscited by 99
  4. 2015Policy Gradient for Coherent Risk Measurescited by 76
  5. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 70
  6. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 40
  7. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  8. 2021Bayesian Distributionally Robust Optimizationcited by 16
  9. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  10. 2021Bayesian Risk Markov Decision Processescited by 6
  11. 2018Risk averse stochastic programming: time consistency and optimal stoppingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Bayesian Dynamic Programming for On-policy Risk-sensitive Reinforcement Learning
  2. 2025An Actor-Critic Algorithm with Function Approximation for Risk Sensitive Cost Markov Decision Processes
  3. 2025Planning and Learning in Average Risk-aware MDPs
  4. 2026Operator-Theoretic Foundations and Policy Gradient Methods for General MDPs with Unbounded Costs
  5. 2024Offline Bayesian Aleatoric and Epistemic Uncertainty Quantification and Posterior Value Optimisation in Finite-State MDPs
  6. 2026Policy Gradient for Continuous-Time Robust Markov Decision Processes
  7. 2024Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action
  8. 2026Risk-Aware General-Utility Markov Decision Processes