mlsanityData to 7 Oct 2026

The Role of Baselines in Policy Gradient Optimization

arXiv 2301.06276 ↗First posted 16 Jan 2023cs.LG cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

We study the effect of baselines in on-policy stochastic policy gradient optimization, and close the gap between the theory and practice of policy optimization methods. Our first contribution is to show that the state value baseline allows on-policy stochastic natural policy gradient (NPG) to converge to a globally optimal policy at an O(1/t) rate, which was not previously known. The analysis relies on two novel findings: the expected progress of the NPG update satisfies a stochastic version of the non-uniform Lojasiewicz (NL) inequality, and with probability 1 the state value baseline prevents the optimal action's probability from vanishing, thus ensuring sufficient exploration. Importantly, these results provide a new understanding of the role of baselines in stochastic policy gradient: by showing that the variance of natural policy gradient estimates remains unbounded with or without a baseline, we find that variance reduction cannot explain their utility in this setting. Instead, the analysis reveals that the primary effect of the value baseline is to reduce the aggressiveness of the updates rather than their variance. That is, we demonstrate that a finite variance is not necessary for almost sure convergence of stochastic NPG, while controlling update aggressiveness is both necessary and sufficient. Additional experimental results verify these theoretical findings.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 17 of its 37 references that are in the map. First posted on arXiv 16 Jan 2023.

Builds on

17 of 37 references.

References · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  5. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  6. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 149
  7. 2020Sample Efficient Reinforcement Learning with REINFORCEcited by 85
  8. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 67
  9. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 65
  10. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  11. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 46
  12. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 46
5 more
  1. 2021Leveraging Non-uniformity in First-order Non-convex Optimizationcited by 44
  2. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  3. 2018Variance Reduction for Reinforcement Learning in Input-Driven Environmentscited by 30
  4. 2021Understanding the Effect of Stochasticity in Policy Optimizationcited by 18
  5. 2020Regret Analysis of a Markov Policy Gradient Algorithm for Multi-arm Banditscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 8; 2026: 7.

25 papers.

Sort:

By area:

  1. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  2. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  3. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  5. 2025TreeRL: LLM Reinforcement Learning with On-Policy Tree Searchcited by 75
  6. 2024QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCEcited by 13
  7. 2024Stochastic Gradient Succeeds for Banditscited by 10
  8. 2024Policy Gradient with Active Importance Samplingcited by 9
  9. 2024Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPscited by 7
  10. 2025Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning ratescited by 4
  11. 2025Enhancing PPO with Trajectory-Aware Hybrid Policiescited by 4
  12. 2026Non-Uniform Noise-to-Signal Ratio in the REINFORCE Policy-Gradient Estimatorcited by 4
13 more
  1. 2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
  2. 2023Accelerated Policy Gradient: On the Convergence Rates of the Nesterov Momentum for Reinforcement Learningcited by 2
  3. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  4. 2023High-performance real-world optical computing trained by in situ gradient-based model-free optimizationcited by 1
  5. 2025Ordering-based Conditions for Global Convergence of Policy Gradient Methodscited by 1
  6. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalizationcited by 1
  7. 2026Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-step Policy Gradientscited by 1
  8. 2023Acceleration in Policy Optimizationcited by 0
  9. 2025Breaking Habits: On the Role of the Advantage Function in Learning Causal State Representationscited by 0
  10. 2025Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learningcited by 0
  11. 2026How Log-Barrier Helps Exploration in Policy Optimizationcited by 0
  12. 2026Concave Statistical Utility Maximization Bandits via Influence-Function Gradientscited by 0
  13. 2026Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learningcited by 0