Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality
Not on the map. Why?
Abstract (arXiv)
A novel Policy Gradient (PG) algorithm, called textitMatryoshka Policy Gradient (MPG), is introduced and studied, in the context of fixed-horizon max-entropy reinforcement learning, where an agent aims at maximizing entropy bonuses additional to its cumulative rewards. In the linear function approximation setting with softmax policies, we prove uniqueness and characterize the optimal policy of the entropy regularized objective, together with global convergence of MPG. These results are proved in the case of continuous state and action space. MPG is intuitive, theoretically sound and we furthermore show that the optimal policy of the infinite horizon max-entropy objective can be approximated arbitrarily well by the optimal policy of the MPG framework. Finally, we provide a criterion for global optimality when the policy is parametrized by a neural network in terms of the neural tangent kernel at convergence. As a proof of concept, we evaluate numerically MPG on standard test benchmarks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 40 of its 62 references that are in the map. First posted on arXiv 22 Mar 2023.
Builds on
40 of 62 references.
References · 40
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2015Trust Region Policy Optimizationcited by 2,580
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2016OpenAI Gymcited by 1,736
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
- 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
28 more
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
- 2019A Theory of Regularized Markov Decision Processescited by 286
- 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
- 2018Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulatorcited by 250
- 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
- 2018Understanding the impact of entropy on policy optimizationcited by 176
- 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 176
- 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
- 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
- 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
- 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
- 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
- 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 86
- 2020Implicit Regularization of Random Feature Modelscited by 54
- 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
- 2020Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizoncited by 50
- 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 48
- 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
- 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
- 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
- 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
- 2016Combining policy gradient and Q-learningcited by 24
- 2022Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regimecited by 21
- 2020Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regimecited by 17
- 2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16
- 2019Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learningcited by 13
- 2022A policy gradient approach for Finite Horizon Constrained Markov Decision Processescited by 8
- 2021Finite Horizon Q-learning: Stability, Convergence, Simulations and an application on Smart Gridscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.