mlsanityData to 7 Oct 2026

Regularized Policies are Reward Robust

arXiv 2101.07012 ↗First posted 18 Jan 2021cs.LG stat.MLcited by 32

Not on the map. Why?

Abstract (arXiv)

Entropic regularization of policies in Reinforcement Learning (RL) is a commonly used heuristic to ensure that the learned policy explores the state-space sufficiently before overfitting to a local optimal policy. The primary motivation for using entropy is for exploration and disambiguating optimal policies; however, the theoretical effects are not entirely understood. In this work, we study the more general regularized RL objective and using Fenchel duality; we derive the dual problem which takes the form of an adversarial reward problem. In particular, we find that the optimal policy found by a regularized objective is precisely an optimal policy of a reinforcement learning problem under a worst-case adversarial reward. Our result allows us to reinterpret the popular entropic regularization scheme as a form of robustification. Furthermore, due to the generality of our results, we apply to other existing regularization schemes. Our results thus give insights into the effects of regularization of policies and deepen our understanding of exploration through robust rewards at large.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 18 of its 44 references that are in the map. First posted on arXiv 18 Jan 2021.

Builds on

18 of 44 references.

References · 18

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016Generative Adversarial Imitation Learningcited by 1,576
  3. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  4. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  5. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  6. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  7. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 182
  8. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  9. 2018Understanding the impact of entropy on policy optimizationcited by 171
  10. 2018Learning to Walk via Deep Reinforcement Learningcited by 153
  11. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  12. 2019Distributionally Robust Optimization and Generalization in Kernel Methodscited by 73
6 more
  1. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 70
  2. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  3. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
  4. 2020Regularized Optimal Transport is Ground Cost Adversarialcited by 17
  5. 2020Distributional Robustness with IPMs and links to Regularization and GANscited by 17
  6. 2020Generalised Lipschitz Regularisation Equals Distributional Robustnesscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 8; 2024: 5; 2025: 4; 2026: 7.

32 papers.

Sort:

By area:

  1. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  2. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  3. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
  4. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  5. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  6. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
  7. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
  8. 2022On the convex formulations of robust Markov decision processescited by 8
  9. 2023Solving Non-Rectangular Reward-Robust MDPs via Frequency Regularizationcited by 8
  10. 2021Model-Free Risk-Sensitive Reinforcement Learningcited by 7
  11. 2022Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularizationcited by 7
  12. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
20 more
  1. 2021ShinRL: A Library for Evaluating RL Algorithms from Theoretical and Practical Perspectivescited by 5
  2. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  3. 2023Policy Gradient Algorithms Implicitly Optimize by Continuationcited by 4
  4. 2023Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practicecited by 4
  5. 2024Behind the Myth of Exploration in Policy Gradientscited by 3
  6. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3
  7. 2025Discrete Compositional Generation via General Soft Operators and Robust Reinforcement Learningcited by 2
  8. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 2
  9. 2023Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularizationcited by 1
  10. 2023Decision-Focused Model-based Reinforcement Learning for Reward Transfercited by 1
  11. 2024Bootstrapping Expectiles in Reinforcement Learningcited by 1
  12. 2024Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraintscited by 1
  13. 2023Bridging the Gap between Newton-Raphson Method and Regularized Policy Iterationcited by 0
  14. 2025Imitate Optimal Policy: Prevail and Induce Action Collapse in Policy Gradientcited by 0
  15. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measurescited by 0
  16. 2026Bounded-Rationality, Hedging, and Generalizationcited by 0
  17. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  18. 2026Generalised Bellman recurrence and three dualities in sequential decision-makingcited by 0
  19. 2026Self-Balancing Sequential Sampling: Fast Convergence with Controlled Predictabilitycited by 0
  20. 2026Fast Regularized Policy Mirror Descent with One-Step TD Updatescited by 0