mlsanityData to 14 Oct 2026

Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity

arXiv 2306.11626 ↗First posted 20 Jun 2023math.OC cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Robust Markov Decision Processes (MDPs) and risk-sensitive MDPs are both powerful tools for making decisions in the presence of uncertainties. Previous efforts have aimed to establish their connections, revealing equivalences in specific formulations. This paper introduces a new formulation for risk-sensitive MDPs, which assesses risk in a slightly different manner compared to the classical Markov risk measure (Ruszczy'nski 2010), and establishes its equivalence with a class of soft robust MDP (RMDP) problems, including the standard RMDP as a special case. Leveraging this equivalence, we further derive the policy gradient theorem for both problems, proving gradient domination and global convergence of the exact policy gradient method under the tabular setting with direct parameterization. This forms a sharp contrast to the Markov risk measure, known to be potentially non-gradient-dominant (Huang et al. 2021). We also propose a sample-based offline learning algorithm, namely the robust fitted-Z iteration (RFZI), for a specific soft RMDP problem with a KL-divergence regularization term (or equivalently the risk-sensitive MDP with an entropy risk measure). We showcase its streamlined design and less stringent assumptions due to the equivalence and analyze its sample complexity

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 47 of its 104 references that are in the map. First posted on arXiv 20 Jun 2023.

Builds on

47 of 104 references.

References · 47

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Robust Adversarial Reinforcement Learningcited by 451
  3. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  4. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  6. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 238
  7. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  8. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  9. 2013Risk-sensitive Reinforcement Learningcited by 130
  10. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  11. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  12. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
35 more
  1. 2022Robust Reinforcement Learning using Offline Datacited by 96
  2. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 87
  3. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  4. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 83
  5. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 78
  6. 2015Policy Gradient for Coherent Risk Measurescited by 76
  7. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  8. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
  9. 2017Reinforcement Learning under Model Mismatchcited by 52
  10. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 52
  11. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 47
  12. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  13. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  14. 2020Distributional Robustness and Regularization in Reinforcement Learningcited by 40
  15. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 38
  16. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  17. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
  18. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 33
  19. 2020Learning with Safety Constraints: Sample Complexity of Reinforcement Learning for Constrained MDPscited by 32
  20. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 32
  21. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 31
  22. 2020Improving Robustness via Risk Averse Distributional Reinforcement Learningcited by 28
  23. 2021Reinforcement Learning with Dynamic Convex Risk Measurescited by 21
  24. 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 20
  25. 2020Markov Decision Processes with Recursive Risk Measurescited by 18
  26. 2021Robust Risk-Aware Reinforcement Learningcited by 18
  27. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 16
  28. 2022Robust Constrained Reinforcement Learningcited by 15
  29. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  30. 2020Distributionally Robust Markov Decision Processes and their Connection to Risk Measurescited by 14
  31. 2020Risk-Averse Learning by Temporal Difference Methodscited by 13
  32. 2021Exploring the Training Robustness of Distributional Reinforcement Learning against Noisy State Observationscited by 12
  33. 2022On the convex formulations of robust Markov decision processescited by 10
  34. 2023On the Global Convergence of Risk-Averse Natural Policy Gradient Methods with Expected Conditional Risk Measurescited by 7
  35. 2018Risk averse stochastic programming: time consistency and optimal stoppingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 4.

17 papers.

Sort:

By area:

  1. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 25
  2. 2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learningcited by 16
  3. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularizationcited by 8
  4. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  5. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 6
  6. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Datacited by 4
  7. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 4
  8. 2025Convergent Q-Learning for Infinite-Horizon General-Sum Markov Games through Behavioral Economicscited by 2
  9. 2025Online Robust Reinforcement Learning with General Function Approximationcited by 2
  10. 2026Actor-Critic Algorithm for Dynamic Expectile and CVaRcited by 2
  11. 2025Optimism as Risk-Seeking in Multi-Agent Reinforcement Learningcited by 1
  12. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
5 more
  1. 2025Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Setscited by 0
  2. 2025Unifying Entropy Regularization in Optimal Control: From and Back to Classical Objectives via Iterated Soft Policies and Path Integral Solutionscited by 0
  3. 2026MDP Planning as Policy Inferencecited by 0
  4. 2026Online Policy Evaluation for MDPs with Dynamic UBSR Measurescited by 0
  5. 2026Bellman-Certified Rounding for Sparse Policy Deployment in MDPscited by 0