A Large Deviations Perspective on Policy Gradient Algorithms
Not on the map. Why?
Abstract (arXiv)
Motivated by policy gradient methods in the context of reinforcement learning, we identify a large deviation rate function for the iterates generated by stochastic gradient descent for possibly non-convex objectives satisfying a Polyak-Lojasiewicz condition. Leveraging the contraction principle from large deviations theory, we illustrate the potential of this result by showing how convergence properties of policy gradient with a softmax parametrization and an entropy regularized objective can be naturally extended to a wide spectrum of other policy parametrizations.
Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 13 Nov 2023.
Builds on
20 of 36 references.
References · 20
- 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 718
- 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 529
- 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 260
- 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 212
- 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
- 2017A unified view of entropy-regularized Markov decision processescited by 177
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 171
- 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 151
- 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
- 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
- 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 105
- 2018Tight Analyses for Non-Smooth Stochastic Gradient Descentcited by 95
8 more
- 2020Variance-Reduced Methods for Machine Learningcited by 79
- 2023High Probability Convergence of Stochastic Gradient Methodscited by 48
- 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 47
- 2020High Probability Convergence Bounds for Non-convex Stochastic Gradient Descent with Sub-Weibull Noisecited by 31
- 2015An Exploration of Softmax Alternatives Belonging to the Spherical Loss Familycited by 27
- 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
- 2022Large deviations rates for stochastic gradient descent with strongly convex functionscited by 9
- 2023Importance Sampling for Stochastic Gradient Descent in Deep Neural Networkscited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.