mlsanityData to 7 Oct 2026

Your Policy Regularizer is Secretly an Adversary

arXiv 2203.12592 ↗First posted 23 Mar 2022cs.LG stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

Policy regularization methods such as maximum entropy regularization are widely used in reinforcement learning to improve the robustness of a learned policy. In this paper, we show how this robustness arises from hedging against worst-case perturbations of the reward function, which are chosen from a limited set by an imagined adversary. Using convex duality, we characterize this robust set of adversarial reward perturbations under KL and alpha-divergence regularization, which includes Shannon and Tsallis entropy regularization as special cases. Importantly, generalization guarantees can be given within this robust set. We provide detailed discussion of the worst-case reward perturbations, and present intuitive empirical examples to illustrate this robustness and its relationship with generalization. Finally, we discuss how our analysis complements and extends previous results on adversarial reward robustness and path consistency optimality conditions.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 31 of its 66 references that are in the map. First posted on arXiv 23 Mar 2022.

Builds on

31 of 66 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2021Convex optimizationcited by 2,479
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 633
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  11. 2019A Theory of Regularized Markov Decision Processescited by 279
  12. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
19 more
  1. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  2. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  3. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  4. 2017A unified view of entropy-regularized Markov decision processescited by 174
  5. 2018Understanding the impact of entropy on policy optimizationcited by 171
  6. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  7. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  8. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  9. 2021Reward is enough for convex MDPscited by 82
  10. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  11. 2017Sparse Markov Decision Processes with Causal Sparse Tsallis Entropy Regularization for Reinforcement Learningcited by 36
  12. 2020Logistic Q-Learningcited by 34
  13. 2021Regularized Policies are Reward Robustcited by 32
  14. 2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
  15. 2019Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcited by 18
  16. 2019Entropic Regularization of Markov Decision Processescited by 15
  17. 2019Multitask Soft Option Learningcited by 13
  18. 2021Variational Inference MPC using Tsallis Divergencecited by 11
  19. 2014An Adversarial Interpretation of Information-Theoretic Bounded Rationalitycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 4; 2025: 5; 2026: 5.

21 papers.

Sort:

By area:

  1. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
  2. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
  3. 2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
  4. 2023Solving Non-Rectangular Reward-Robust MDPs via Frequency Regularizationcited by 8
  5. 2022Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularizationcited by 7
  6. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6
  7. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  8. 2026Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectificationcited by 5
  9. 2023Policy Gradient Algorithms Implicitly Optimize by Continuationcited by 4
  10. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learningcited by 4
  11. 2024Behind the Myth of Exploration in Policy Gradientscited by 3
  12. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3
9 more
  1. 2025Discrete Compositional Generation via General Soft Operators and Robust Reinforcement Learningcited by 2
  2. 2022Entropy Regularization for Population Estimationcited by 1
  3. 2023Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularizationcited by 1
  4. 2024Bootstrapping Expectiles in Reinforcement Learningcited by 1
  5. 2025HAEPO: History-Aggregated Exploratory Policy Optimizationcited by 0
  6. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measurescited by 0
  7. 2026Bounded-Rationality, Hedging, and Generalizationcited by 0
  8. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0
  9. 2026Generalised Bellman recurrence and three dualities in sequential decision-makingcited by 0