mlsanityData to 14 Oct 2026

Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraints

arXiv 2411.19193 ↗First posted 28 Nov 2024cs.LG cs.AI math.OCcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper examines reinforcement learning (RL) in infinite-horizon decision processes with almost-sure safety constraints, crucial for applications like autonomous systems, finance, and resource management. We propose a doubly-regularized RL framework combining reward and parameter regularization to address safety constraints in continuous state-action spaces. The problem is formulated as a convex regularized objective with parametrized policies in the mean-field regime. Leveraging mean-field theory and Wasserstein gradient flows, policies are modeled on an infinite-dimensional statistical manifold, with updates governed by parameter distribution gradient flows. Key contributions include solvability conditions for safety-constrained problems, smooth bounded approximations for gradient flows, and exponential convergence guarantees under sufficient regularization. General regularization conditions, including entropy regularization, support practical particle method implementations. This framework provides robust theoretical insights and guarantees for safe RL in complex, high-dimensional settings.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 49 references that are in the map. First posted on arXiv 28 Nov 2024.

Builds on

33 of 49 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  6. 2019Control Barrier Functions: Theory and Applicationscited by 372
  7. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  8. 2019A Theory of Regularized Markov Decision Processescited by 286
  9. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  10. 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
  11. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  12. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
21 more
  1. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  2. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  3. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  4. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
  5. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 69
  6. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  7. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  8. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 44
  9. 2021Regularized Policies are Reward Robustcited by 32
  10. 2021Feasible Actor-Critic: Constrained Reinforcement Learning for Ensuring Statewise Safetycited by 30
  11. 2021Joint Synthesis of Safety Certificate and Safe Control Policy using Constrained Reinforcement Learningcited by 27
  12. 2023A Fisher-Rao gradient flow for entropy-regularised Markov decision processes in Polish spacescited by 23
  13. 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
  14. 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17
  15. 2021Linear Convergence of Entropy-Regularized Natural Policy Gradient with Linear Function Approximationcited by 16
  16. 2014Control to flocking of the kinetic Cucker-Smale modelcited by 6
  17. 2022Effects of Safety State Augmentation on Safe Explorationcited by 6
  18. 2017The Pontryagin Maximum Principle in the Wasserstein Spacecited by 3
  19. 2023Some Convexity Criteria for Differentiable Functions on the 2-Wasserstein Spacecited by 3
  20. 2022Epi-Convergence of Expectation Functions under Varying Measures and Integrandscited by 2
  21. 2024Mean-Field Control Barrier Functions: A Framework for Real-Time Swarm Controlcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Non-convex entropic mean-field optimization via Best Response flowcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization
  2. 2026Provably Safe Reinforcement Learning for Stochastic Reach-Avoid Problems with Entropy Regularization
  3. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning
  4. 2026Constrained Flow Policy Updates: A Generalized Schr"odinger Bridge View
  5. 2026Reinforcement Learning in Real Option Models
  6. 2024Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems
  7. 2025State Entropy Regularization for Robust Reinforcement Learning
  8. 2025Anytime Safe Reinforcement Learning