mlsanityData to 7 Oct 2026

RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Risk

arXiv 2209.04067 ↗First posted 9 Sep 2022cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Prior work on safe Reinforcement Learning (RL) has studied risk-aversion to randomness in dynamics (aleatory) and to model uncertainty (epistemic) in isolation. We propose and analyze a new framework to jointly model the risk associated with epistemic and aleatory uncertainties in finite-horizon and discounted infinite-horizon MDPs. We call this framework that combines Risk-Averse and Soft-Robust methods RASR. We show that when the risk-aversion is defined using either EVaR or the entropic risk, the optimal policy in RASR can be computed efficiently using a new dynamic program formulation with a time-dependent risk level. As a result, the optimal risk-averse policies are deterministic but time-dependent, even in the infinite-horizon discounted setting. We also show that particular RASR objectives reduce to risk-averse RL with mean posterior transition probabilities. Our empirical results show that our new algorithms consistently mitigate uncertainty as measured by EVaR and other standard risk measures.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 20 of its 71 references that are in the map. First posted on arXiv 9 Sep 2022.

Builds on

20 of 71 references.

References · 20

  1. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  2. 2019A Theory of Regularized Markov Decision Processescited by 279
  3. 2017A unified view of entropy-regularized Markov decision processescited by 174
  4. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 162
  5. 2014Algorithms for CVaR Optimization in MDPscited by 98
  6. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 76
  7. 2015Policy Gradient for Coherent Risk Measurescited by 74
  8. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 57
  9. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 49
  10. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  11. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  12. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
8 more
  1. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  2. 2019Entropic Risk Measure in Policy Searchcited by 19
  3. 2019Epistemic Risk-Sensitive Reinforcement Learningcited by 17
  4. 2020First-Order Methods for Wasserstein Distributionally Robust MDPcited by 16
  5. 2020Constrained Risk-Averse Markov Decision Processescited by 15
  6. 2021Policy Gradient Bayesian Robust Optimization for Imitation Learningcited by 11
  7. 2020Risk-Sensitive Motion Planning using Entropic Value-at-Riskcited by 7
  8. 2021Risk-Averse Decision Making Under Uncertaintycited by 2

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2023Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theorycited by 5
  2. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 3
  3. 2023On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processescited by 0
  4. 2023DRL-ORA: Distributional Reinforcement Learning with Online Risk Adaptioncited by 0