mlsanityData to 14 Oct 2026

State Entropy Regularization for Robust Reinforcement Learning

arXiv 2506.07085 ↗First posted 8 Jun 2025cs.LG stat.MLcited by 6

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its theoretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus on small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, as well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy are more sensitive to the number of rollouts used for policy evaluation.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 69 references that are in the map. First posted on arXiv 8 Jun 2025.

Builds on

46 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measures9 Dec
  2. 2024How to Explore with Belief: State Entropy Maximization in POMDPs4 Jun

Other references · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  5. 2017Robust Adversarial Reinforcement Learningcited by 451
  6. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  7. 2019Efficient Exploration via State Marginal Matchingcited by 188
  8. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
  9. 2017A unified view of entropy-regularized Markov decision processescited by 182
  10. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  11. 2021Reinforcement Learning with Prototypical Representationscited by 179
  12. 2018Understanding the impact of entropy on policy optimizationcited by 176
32 more
  1. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  2. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  3. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  4. 2019Robust Learning with Jacobian Regularizationcited by 129
  5. 2021APS: Active Pretraining with Successor Featurescited by 112
  6. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 108
  7. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  8. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 47
  9. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  10. 2023Fast Rates for Maximum Entropy Explorationcited by 33
  11. 2021Geometric Entropic Explorationcited by 32
  12. 2021Regularized Policies are Reward Robustcited by 32
  13. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
  14. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  15. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  16. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  17. 2019An Intrinsically-Motivated Approach for Learning Highly Exploring and Fast Mixing Policiescited by 20
  18. 2020Active Model Estimation in Markov Decision Processescited by 20
  19. 2023Explore to Generalize in Zero-Shot RLcited by 20
  20. 2022R'enyi State Entropy for Exploration Acceleration in Reinforcement Learningcited by 18
  21. 2020Distributional Robustness with IPMs and links to Regularization and GANscited by 17
  22. 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
  23. 2019Marginalized State Distribution Entropy Regularization in Policy Optimizationcited by 16
  24. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  25. 2022k-Means Maximum Entropy Explorationcited by 11
  26. 2015On the Dual Representation of Coherent Risk Measurescited by 8
  27. 2023Solving Non-Rectangular Reward-Robust MDPs via Frequency Regularizationcited by 8
  28. 2016Analysis of k-Nearest Neighbor Distances with Application to Entropy Estimationcited by 6
  29. 2024The Limits of Pure Exploration in POMDPs: When the Observation Entropy is Enoughcited by 6
  30. 2023A Tale of Sampling and Estimation in Discounted Reinforcement Learningcited by 4
  31. 2025Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processescited by 3
  32. 2024ELEMENT: Episodic and Lifelong Exploration via Maximum Entropycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  2. 2025Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruningcited by 1
  3. 2025Stress-Aware Learning under KL Drift via Trust-Decayed Mirror Descentcited by 0
  4. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  5. 2026TRE: Encouraging Exploration in the Trust Regioncited by 0
  6. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization
  2. 2021Robust Entropy-regularized Markov Decision Processes
  3. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning
  4. 2026Provably Safe Reinforcement Learning for Stochastic Reach-Avoid Problems with Entropy Regularization
  5. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularization
  6. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
  7. 2026Robust Regularized Policy Iteration under Transition Uncertainty
  8. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency