mlsanityData to 7 Oct 2026

PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation

arXiv 2202.00308 ↗First posted 1 Feb 2022cs.LG math.OCcited by 15

Not on the map. Why?

Abstract (arXiv)

Despite their success, policy gradient methods suffer from high variance of the gradient estimate, which can result in unsatisfactory sample complexity. Recently, numerous variance-reduced extensions of policy gradient methods with provably better sample complexity and competitive numerical performance have been proposed. After a compact survey on some of the main variance-reduced REINFORCE-type methods, we propose ProbAbilistic Gradient Estimation for Policy Gradient (PAGE-PG), a novel loopless variance-reduced policy gradient method based on a probabilistic switch between two types of updates. Our method is inspired by the PAGE estimator for supervised learning and leverages importance sampling to obtain an unbiased gradient estimator. We show that PAGE-PG enjoys a Oleft( ε-3 right) average sample complexity to reach an ε-stationary solution, which matches the sample complexity of its most competitive counterparts under the same setting. A numerical evaluation confirms the competitive performance of our method on classical control tasks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

25 of 34 references.

References · 25

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2016OpenAI Gymcited by 1,684
  6. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  7. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  8. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  9. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 218
  10. 2018Understanding the impact of entropy on policy optimizationcited by 171
  11. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  12. 2016Katyusha: The First Direct Acceleration of Stochastic Gradient Methodscited by 134
13 more
  1. 2020PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimizationcited by 116
  2. 2018Stochastic Variance-Reduced Policy Gradientcited by 103
  3. 2019Don't Jump Through Hoops and Remove Those Loops: SVRG and Katyusha are Better Without the Outer Loopcited by 68
  4. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  5. 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 58
  6. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 58
  7. 2017Variance Reduced Value Iteration and Faster Algorithms for Solving Markov Decision Processescited by 56
  8. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 51
  9. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  10. 2020Momentum-Based Policy Gradient Methodscited by 28
  11. 2020Stochastic Recursive Momentum for Policy Gradient Methodscited by 22
  12. 2020On the Promise of the Stochastic Generalized Gauss-Newton Method for Training DNNscited by 18
  13. 2019Towards Understanding the Importance of Noise in Training Neural Networkscited by 10

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 7; 2025: 3.

15 papers.

Sort:

  1. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
  2. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 24
  3. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  4. 2022Momentum-Based Policy Gradient with Second-Order Informationcited by 11
  5. 2024Sample-Efficient Constrained Reinforcement Learning with General Parameterizationcited by 11
  6. 2024Policy Gradient with Active Importance Samplingcited by 9
  7. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 7
  8. 2024Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reductioncited by 4
  9. 2024Decentralized Federated Policy Gradient with Byzantine Fault-Tolerance and Provably Fast Convergencecited by 2
  10. 2025Non-Asymptotic Global Convergence of PPO-Clipcited by 2
  11. 2023Efficiently Escaping Saddle Points for Policy Optimizationcited by 1
  12. 2024On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimizationcited by 1
3 more
  1. 2025Reusing Trajectories in Policy Gradients Enables Fast Convergencecited by 1
  2. 2024Last-Iterate Convergence of General Parameterized Policies in Constrained MDPscited by 0
  3. 2025A Variance-Reduced Cubic-Regularized Newton for Policy Optimizationcited by 0