mlsanityData to 7 Oct 2026

Risk-Averse MDPs under Reward Ambiguity

arXiv 2301.01045 ↗First posted 3 Jan 2023cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose a distributionally robust return-risk model for Markov decision processes (MDPs) under risk and reward ambiguity. The proposed model optimizes the weighted average of mean and percentile performances, and it covers the distributionally robust MDPs and the distributionally robust chance-constrained MDPs (both under reward ambiguity) as special cases. By considering that the unknown reward distribution lies in a Wasserstein ambiguity set, we derive the tractable reformulation for our model. In particular, we show that that the return-risk model can also account for risk from uncertain transition kernel when one only seeks deterministic policies, and that a distributionally robust MDP under the percentile criterion can be reformulated as its nominal counterpart at an adjusted risk level. A scalable first-order algorithm is designed to solve large-scale problems, and we demonstrate the advantages of our proposed model and algorithm through numerical experiments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 87 references that are in the map. First posted on arXiv 3 Jan 2023.

Builds on

25 of 87 references.

References · 25

  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2019Model-Based Reinforcement Learning for Ataricited by 495
  3. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  4. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  6. 2016Distributionally Robust Stochastic Optimization with Wasserstein Distancecited by 182
  7. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  8. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  9. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  10. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  11. 2017Reinforcement Learning under Model Mismatchcited by 52
  12. 2019Wasserstein Robust Reinforcement Learningcited by 50
13 more
  1. 2020Distributional Robustness and Regularization in Reinforcement Learningcited by 39
  2. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 32
  3. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  4. 2018Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approachcited by 27
  5. 2021Robust Risk-Aware Reinforcement Learningcited by 17
  6. 2020First-Order Methods for Wasserstein Distributionally Robust MDPcited by 16
  7. 2018On Distributionally Robust Chance Constrained Programs with Wasserstein Distancecited by 15
  8. 2020Constrained Risk-Averse Markov Decision Processescited by 15
  9. 2020Minimizing Spectral Risk Measures Applied to Markov Decision Processescited by 12
  10. 2018Data-Driven Chance Constrained Programs over Wasserstein Ballscited by 9
  11. 2016Risk Aversion in Finite Markov Decision Processes Using Total Cost Criteria and Average Value at Riskcited by 8
  12. 2015Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measurescited by 5
  13. 2016Optimizing Quantiles in Preference-based Markov Decision Processescited by 2

Built on by

Not cited yet.