mlsanityData to 7 Oct 2026

Stochastic Variance-Reduced Policy Gradient

arXiv 1806.05618 ↗First posted 14 Jun 2018cs.LG stat.MLcited by 103

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel reinforcement- learning algorithm consisting in a stochastic variance-reduced version of policy gradient for solving Markov Decision Processes (MDPs). Stochastic variance-reduced gradient (SVRG) methods have proven to be very successful in supervised learning. However, their adaptation to policy gradient is not straightforward and needs to account for I) a non-concave objective func- tion; II) approximations in the full gradient com- putation; and III) a non-stationary sampling pro- cess. The result is SVRPG, a stochastic variance- reduced policy gradient algorithm that leverages on importance weights to preserve the unbiased- ness of the gradient estimate. Under standard as- sumptions on the MDP, we provide convergence guarantees for SVRPG with a convergence rate that is linear under increasing batch sizes. Finally, we suggest practical variants of SVRPG, and we empirically evaluate them on continuous MDPs.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 16; 2023: 22; 2024: 19; 2025: 14; 2026: 12.

103 papers.

Sort:

By area:

  1. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  2. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  3. 2021A general sample complexity analysis of vanilla policy gradientcited by 74
  4. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  5. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  6. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
  7. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
  8. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 44
  9. 2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
  10. 2022Finite-time analysis of single-timescale actor-criticcited by 30
  11. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 24
  12. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 23
38 more
  1. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  2. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 20
  3. 2022Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithmcited by 19
  4. 2021Accelerated and instance-optimal policy evaluation with linear function approximationcited by 18
  5. 2023Beyond Exponentially Fast Mixing in Average-Reward Reinforcement Learning via Multi-Level Monte Carlo Actor-Criticcited by 18
  6. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  7. 2021Bregman Gradient Policy Optimizationcited by 15
  8. 2022PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimationcited by 15
  9. 2024A Sharper Global Convergence Analysis for Average Reward Reinforcement Learning via an Actor-Critic Approachcited by 14
  10. 2021Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithmcited by 12
  11. 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
  12. 2022On the Hidden Biases of Policy Mirror Ascent in Continuous Action Spacescited by 11
  13. 2022Momentum-Based Policy Gradient with Second-Order Informationcited by 11
  14. 2024FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHFcited by 11
  15. 2021Truncated Emphatic Temporal Difference Methods for Prediction and Controlcited by 10
  16. 2024Learning Optimal Deterministic Policies with Stochastic Policy Gradientscited by 10
  17. 2022Provably Efficient Fictitious Play Policy Optimization for Zero-Sum Markov Games with Structured Transitionscited by 9
  18. 2023Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methodscited by 9
  19. 2024Policy Gradient with Active Importance Samplingcited by 9
  20. 2021Factored Policy Gradients: Leveraging Structure for Efficient Learning in MOMDPscited by 8
  21. 2023Regret-Optimal Model-Free Reinforcement Learning for Discounted MDPs with Short Burn-In Timecited by 8
  22. 2025Global Convergence for Average Reward Constrained MDPs with Primal-Dual Actor Critic Algorithmcited by 8
  23. 2021Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methodscited by 7
  24. 2023A Cubic-regularized Policy Newton Algorithm for Reinforcement Learningcited by 7
  25. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 7
  26. 2021Policy Gradient Methods for Distortion Risk Measurescited by 6
  27. 2021MDPGT: Momentum-based Decentralized Policy Gradient Trackingcited by 6
  28. 2023FedHQL: Federated Heterogeneous Q-Learningcited by 6
  29. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversariescited by 6
  30. 2024Towards Global Optimality for Practical Average Reward Reinforcement Learning without Mixing Time Oraclescited by 6
  31. 2024On Divergence Measures for Training GFlowNetscited by 6
  32. 2021Smoothed functional-based gradient algorithms for off-policy reinforcement learning: A non-asymptotic viewpointcited by 5
  33. 2023On the Global Convergence of Natural Actor-Critic with Two-layer Neural Network Parametrizationcited by 5
  34. 2023On the Second-Order Convergence of Biased Policy Gradient Algorithmscited by 5
  35. 2024Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysiscited by 5
  36. 2021On the Convergence Rate of Off-Policy Policy Optimization Methods with Density-Ratio Correctioncited by 4
  37. 2022Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimizationcited by 4
  38. 2024Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reductioncited by 4

50 most cited of 103 All 103 (JSON)