mlsanityData to 14 Oct 2026

Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning

arXiv 2607.03168 ↗First posted 3 Jul 2026math.OC cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Entropy regularization is widely used in continuous-time reinforcement learning (RL) to reduce sensitivity to environmental perturbations, yet its robustness benefits lack a rigorous theoretical foundation. This paper establishes the first robustness guarantees for entropy-regularized continuous-time Markov decision processes. We show that maximizing an entropy-regularized objective yields a lower bound on a worst-case robust RL problem with joint reward and transition perturbations. We analytically characterize the induced robust sets and prove that they expand monotonically with the regularization strength, justifying the empirical observation that stronger entropy improves robustness. In contrast to prior discrete-time analyses, our results remove the intractable state-distribution entropy term and provide guarantees invariant to action frequency. Experiments on queueing network control and market making confirm our theory, showing that entropy-regularized policies outperform greedy and ε-greedy baselines under dynamics perturbations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 42 of its 62 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

42 of 62 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Continuous-time q-learning for Markov regime switching system under Tsallis entropy27 Jan
  2. 2025Continuous-time reinforcement learning for optimal switching over multiple regimes4 Dec
  3. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State28 Sep
  4. 2025State Entropy Regularization for Robust Reinforcement Learning8 Jun
  5. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learning13 Mar
  6. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty19 Apr

Other references · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  6. 2017Robust Adversarial Reinforcement Learningcited by 451
  7. 2019A Theory of Regularized Markov Decision Processescited by 286
  8. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  9. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  10. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  11. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  12. 2022q-Learning in Continuous Timecited by 91
24 more
  1. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  2. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  3. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 47
  4. 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 43
  5. 2020Queueing Network Controls via Deep Reinforcement Learningcited by 41
  6. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 38
  7. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36
  8. 2021Regularized Policies are Reward Robustcited by 32
  9. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 31
  10. 2023Policy Optimization for Continuous Reinforcement Learningcited by 31
  11. 2020Scalable First-Order Methods for Robust MDPscited by 25
  12. 2022Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problemscited by 25
  13. 2023Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithmscited by 25
  14. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 24
  15. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  16. 2023Actor-Critic learning for mean-field control in continuous timecited by 22
  17. 2023Continuous-time q-learning for mean-field control problemscited by 22
  18. 2022Your Policy Regularizer is Secretly an Adversarycited by 21
  19. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 16
  20. 2020Regularity and stability of feedback relaxed controlscited by 13
  21. 2021Robust Value Iteration for Continuous Control Taskscited by 8
  22. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundationscited by 6
  23. 2024Reinforcement Learning for Intensity Control: An Application to Choice-Based Network Revenue Managementcited by 3
  24. 2025Entropy Regularization in Mean-Field Games of Optimal Stoppingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization
  2. 2021Robust Entropy-regularized Markov Decision Processes
  3. 2026Provably Safe Reinforcement Learning for Stochastic Reach-Avoid Problems with Entropy Regularization
  4. 2024Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraints
  5. 2024Optimal Rates of Convergence for Entropy Regularization in Discounted Markov Decision Processes
  6. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
  7. 2026FERPO: Forward Entropy-Regularized Policy Optimization
  8. 2026A Comparative Theoretical Analysis of Entropy Control Methods in Reinforcement Learning