mlsanityData to 7 Oct 2026

Twice regularized MDPs and the equivalence between robustness and regularization

arXiv 2110.06267 ↗First posted 12 Oct 2021cs.LG math.OCcited by 46

Not on the map. Why?

Abstract (arXiv)

Robust Markov decision processes (MDPs) aim to handle changing or partially known system dynamics. To solve them, one typically resorts to robust optimization methods. However, this significantly increases computational complexity and limits scalability in both learning and planning. On the other hand, regularized MDPs show more stability in policy learning without impairing time complexity. Yet, they generally do not encompass uncertainty in the model dynamics. In this work, we aim to learn robust MDPs using regularization. We first show that regularized MDPs are a particular instance of robust MDPs with uncertain reward. We thus establish that policy iteration on reward-robust MDPs can have the same time complexity as on regularized MDPs. We further extend this relationship to MDPs with uncertain transitions: this leads to a regularization term with an additional dependence on the value function. We finally generalize regularized MDPs to twice regularized MDPs (R2 MDPs), i.e., MDPs with textitboth value and policy regularization. The corresponding Bellman operators enable developing policy iteration schemes with convergence and robustness guarantees. It also reduces planning and learning in robust MDPs to regularized MDPs.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 20 of its 55 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

20 of 55 references.

References · 20

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  4. 2017Robust Adversarial Reinforcement Learningcited by 439
  5. 2019A Theory of Regularized Markov Decision Processescited by 279
  6. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  7. 2019Wasserstein Distributionally Robust Optimization: Theory and Applications in Machine Learningcited by 169
  8. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  9. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  10. 2015Distributionally Robust Logistic Regressioncited by 124
  11. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
  12. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
8 more
  1. 2018Differentiable Dynamic Programming for Structured Prediction and Attentioncited by 58
  2. 2017Reinforcement Learning under Model Mismatchcited by 52
  3. 2020Distributional Robustness and Regularization in Reinforcement Learningcited by 39
  4. 2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
  5. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 36
  6. 2021Regularized Policies are Reward Robustcited by 32
  7. 2020Scalable First-Order Methods for Robust MDPscited by 22
  8. 2018Learning Robust Optionscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 12; 2024: 10; 2025: 9; 2026: 3.

46 papers.

Sort:

By area:

  1. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  2. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  3. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  4. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
  5. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
  6. 2023Maximum entropy GFlowNets with soft Q-learningcited by 30
  7. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
  8. 2023Robust Average-Reward Markov Decision Processescited by 22
  9. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  10. 2023Towards Minimax Optimality of Model-based Robust Reinforcement Learningcited by 21
  11. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
  12. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
34 more
  1. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  2. 2022Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexitycited by 14
  3. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  4. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
  5. 2022On the convex formulations of robust Markov decision processescited by 8
  6. 2023Solving Non-Rectangular Reward-Robust MDPs via Frequency Regularizationcited by 8
  7. 2022Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularizationcited by 7
  8. 2023Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernelcited by 6
  9. 2023Robust Multi-Agent Reinforcement Learning by Mutual Information Regularizationcited by 6
  10. 2024Decoding Game: On Minimax Optimality of Heuristic Text Generation Strategiescited by 6
  11. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  12. 2023An Efficient Solution to s-Rectangular Robust Markov Decision Processescited by 5
  13. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  14. 2022RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Riskcited by 4
  15. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  16. 2025Adversarial Diffusion for Robust Reinforcement Learningcited by 4
  17. 2025Evidence on the Regularisation Properties of Maximum-Entropy Reinforcement Learningcited by 3
  18. 2025Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processescited by 3
  19. 2024RRLS : Robust Reinforcement Learning Suitecited by 2
  20. 2025Model-Free Robust Average-Reward Reinforcement Learning with Sample Complexity Analysiscited by 2
  21. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  22. 2025Discrete Compositional Generation via General Soft Operators and Robust Reinforcement Learningcited by 2
  23. 2022Robust Phi-Divergence MDPscited by 1
  24. 2023Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularizationcited by 1
  25. 2023Decision-Focused Model-based Reinforcement Learning for Reward Transfercited by 1
  26. 2024Regularized Q-Learning with Linear Function Approximationcited by 1
  27. 2024Bootstrapping Expectiles in Reinforcement Learningcited by 1
  28. 2024Decoupling regularization from the action spacecited by 1
  29. 2024Solving robust MDPs as a sequence of static RL problemscited by 1
  30. 2022User-Oriented Robust Reinforcement Learningcited by 0
  31. 2023Where2Start: Leveraging initial States for Robust and Sample-Efficient Reinforcement Learningcited by 0
  32. 2026Bounded-Rationality, Hedging, and Generalizationcited by 0
  33. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  34. 2026Generalised Bellman recurrence and three dualities in sequential decision-makingcited by 0