mlsanityData to 7 Oct 2026

Bregman Gradient Policy Optimization

arXiv 2106.12112 ↗First posted 23 Jun 2021cs.LG cs.RO math.OCcited by 15

Not on the map. Why?

Abstract (arXiv)

In the paper, we design a novel Bregman gradient policy optimization framework for reinforcement learning based on Bregman divergences and momentum techniques. Specifically, we propose a Bregman gradient policy optimization (BGPO) algorithm based on the basic momentum technique and mirror descent iteration. Meanwhile, we further propose an accelerated Bregman gradient policy optimization (VR-BGPO) algorithm based on the variance reduced technique. Moreover, we provide a convergence analysis framework for our Bregman gradient policy optimization under the nonconvex setting. We prove that our BGPO achieves a sample complexity of O(ε-4) for finding ε-stationary policy only requiring one trajectory at each iteration, and our VR-BGPO reaches the best known sample complexity of O(ε-3), which also only requires one trajectory at each iteration. In particular, by using different Bregman divergences, our BGPO framework unifies many existing policy optimization algorithms such as the existing (variance reduced) policy gradient algorithms such as natural policy gradient algorithm. Extensive experimental results on multiple reinforcement learning tasks demonstrate the efficiency of our new algorithms.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 23 Jun 2021.

Builds on

32 of 55 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2016OpenAI Gymcited by 1,684
  7. 2017Deep Reinforcement Learning: An Overviewcited by 327
  8. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  9. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  10. 2018SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimatorcited by 283
  11. 2019A Theory of Regularized Markov Decision Processescited by 279
  12. 2017A unified view of entropy-regularized Markov decision processescited by 174
20 more
  1. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  2. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  3. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  4. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
  5. 2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 113
  6. 2018Stochastic Variance-Reduced Policy Gradientcited by 103
  7. 2020Mirror Descent Policy Optimizationcited by 102
  8. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 85
  9. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 83
  10. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  11. 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 58
  12. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 58
  13. 2020Efficiently Solving MDPs with Stochastic Mirror Descentcited by 52
  14. 2019Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policycited by 42
  15. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  16. 2019Hybrid Stochastic Gradient Descent Algorithms for Stochastic Nonconvex Optimizationcited by 31
  17. 2020Momentum-Based Policy Gradient Methodscited by 28
  18. 2018On the Convergence Rate of Stochastic Mirror Descent for Nonsmooth Nonconvex Optimizationcited by 18
  19. 2019Policy Optimization with Stochastic Mirror Descentcited by 17
  20. 2020A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 2; 2023: 3; 2024: 5; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2021A general sample complexity analysis of vanilla policy gradientcited by 74
  2. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 29
  3. 2021Enhanced Bilevel Optimization via Bregman Distancecited by 28
  4. 2021Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysiscited by 25
  5. 2022Geometry and convergence of natural policy gradient methodscited by 12
  6. 2022Momentum-Based Policy Gradient with Second-Order Informationcited by 11
  7. 2024Taming Nonconvex Stochastic Mirror Descent with General Bregman Divergencecited by 11
  8. 2023Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methodscited by 9
  9. 2024Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignmentcited by 9
  10. 2024Spurious Stationarity and Hardness Results for Bregman Proximal-Type Algorithmscited by 2
  11. 2024Online Nonconvex Bilevel Optimization with Bregman Divergencescited by 2
  12. 2025Policy Mirror Descent with Temporal Difference Learning: Sample Complexity under Online Markov Datacited by 2
3 more
  1. 2023Efficiently Escaping Saddle Points for Policy Optimizationcited by 1
  2. 2024On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimizationcited by 1
  3. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0