mlsanityData to 7 Oct 2026

Policy Gradient for Rectangular Robust Markov Decision Processes

arXiv 2301.13589 ↗First posted 31 Jan 2023cs.LG cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Policy gradient methods have become a standard for training reinforcement learning agents in a scalable and efficient manner. However, they do not account for transition uncertainty, whereas learning robust policies can be computationally expensive. In this paper, we introduce robust policy gradient (RPG), a policy-based method that efficiently solves rectangular robust Markov decision processes (MDPs). We provide a closed-form expression for the worst occupation measure. Incidentally, we find that the worst kernel is a rank-one perturbation of the nominal. Combining the worst occupation measure with a robust Q-value estimation yields an explicit form of the robust gradient. Our resulting RPG can be estimated from data with the same time complexity as its non-robust equivalent. Hence, it relieves the computational burden of convex optimization problems required for training robust policies by current policy gradient approaches.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 19 of its 36 references that are in the map. First posted on arXiv 31 Jan 2023.

Builds on

19 of 36 references.

References · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Robust Adversarial Reinforcement Learningcited by 439
  3. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  4. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  5. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  6. 2015Policy Gradient for Coherent Risk Measurescited by 74
  7. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  8. 2017Reinforcement Learning under Model Mismatchcited by 52
  9. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 49
  10. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  11. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  12. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 36
7 more
  1. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
  2. 2021Regularized Policies are Reward Robustcited by 32
  3. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
  4. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  5. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
  6. 2018Learning Robust Optionscited by 17
  7. 2022Robust Constrained Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 9; 2025: 10; 2026: 6.

29 papers.

Sort:

By area:

  1. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  2. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 24
  3. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meetcited by 18
  4. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
  5. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
  6. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
  7. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processescited by 12
  8. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  9. 2025Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learningcited by 6
  10. 2024Test-Time Regret Minimization in Meta Reinforcement Learningcited by 5
  11. 2025Robust Finite-Memory Policy Gradients for Hidden-Model POMDPscited by 5
  12. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
17 more
  1. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  2. 2023CROP: Conservative Reward for Model-based Offline Policy Optimizationcited by 3
  3. 2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3
  4. 2024Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Modelcited by 3
  5. 2025Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processescited by 3
  6. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  7. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  8. 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learningcited by 2
  9. 2025Multi-Agent Reinforcement Learning in Intelligent Transportation Systems: A Comprehensive Surveycited by 2
  10. 2024Robust Reinforcement Learning with Dynamic Distortion Risk Measurescited by 1
  11. 2025Distributionally Robust Self Paced Curriculum Reinforcement Learningcited by 1
  12. 2026Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterizationcited by 1
  13. 2025Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Setscited by 0
  14. 2026Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RLcited by 0
  15. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  16. 2026Expert Behavior Prior Reinforcement Learningcited by 0
  17. 2026Bellman-Certified Rounding for Sparse Policy Deployment in MDPscited by 0