mlsanityData to 14 Oct 2026

Enhancing Policy Gradient with the Polyak Step-Size Adaption

arXiv 2404.07525 ↗First posted 11 Apr 2024cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Policy gradient is a widely utilized and foundational algorithm in the field of reinforcement learning (RL). Renowned for its convergence guarantees and stability compared to other RL algorithms, its practical application is often hindered by sensitivity to hyper-parameters, particularly the step-size. In this paper, we introduce the integration of the Polyak step-size in RL, which automatically adjusts the step-size without prior knowledge. To adapt this method to RL settings, we address several issues, including unknown f* in the Polyak step-size. Additionally, we showcase the performance of the Polyak step-size in RL through experiments, demonstrating faster convergence and the attainment of more stable policies.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 26 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

13 of 26 references.

References · 13

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2020Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergencecited by 137
  6. 2019Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Ratescited by 119
  7. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  8. 2022Dynamics of SGD with Stochastic Polyak Stepsizes: Truly Adaptive Variants and Convergence to Exact Solutioncited by 40
  9. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 30
  10. 2023MoMo: Momentum Models for Adaptive Learning Ratescited by 24
  11. 2023Adaptive SGD with Polyak stepsize and Line-search: Robust Convergence and Variance Reductioncited by 23
  12. 2021Towards Noise-adaptive, Problem-adaptive (Accelerated) Stochastic Gradient Descentcited by 16
1 more
  1. 2020Visualizing the Loss Landscape of Actor Critic Methods with Applications in Inventory Optimizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Byzantine-Robust Optimization under (L0, L1)-Smoothnesscited by 0
  2. 2026Why Clipping Matters in AdaGrad? Toward a High-Probability Theory under Generalized Smoothnesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations
  2. 2023Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes
  3. 2025Policy Gradient with Tree Search: Avoiding Local Optimas through Lookahead
  4. 2025Residual Policy Gradient: A Reward View of KL-regularized Objective
  5. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  6. 2026Implicit Q-Learning and SARSA: Liberating Policy Control from Step-Size Calibration
  7. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
  8. 2025A Practical Introduction to Deep Reinforcement Learning