mlsanityData to 14 Oct 2026

Heavy-Ball Momentum Accelerated Actor-Critic With Function Approximation

arXiv 2408.06945 ↗First posted 13 Aug 2024cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

By using an parametric value function to replace the Monte-Carlo rollouts for value estimation, the actor-critic (AC) algorithms can reduce the variance of stochastic policy gradient so that to improve the convergence rate. While existing works mainly focus on analyzing convergence rate of AC algorithms under Markovian noise, the impacts of momentum on AC algorithms remain largely unexplored. In this work, we first propose a heavy-ball momentum based advantage actor-critic (mboxHB-A2C) algorithm by integrating the heavy-ball momentum into the critic recursion that is parameterized by a linear function. When the sample trajectory follows a Markov decision process, we quantitatively certify the acceleration capability of the proposed HB-A2C algorithm. Our theoretical results demonstrate that the proposed HB-A2C finds an ε-approximate stationary point with ooε-2 iterations for reinforcement learning tasks with Markovian noise. Moreover, we also reveal the dependence of learning rates on the length of the sample trajectory. By carefully selecting the momentum factor of the critic recursion, the proposed HB-A2C can balance the errors introduced by the initialization and the stoschastic approximation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 20 of its 38 references that are in the map. First posted on arXiv 13 Aug 2024.

Builds on

20 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces10 Jan

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  3. 2021Policy Mirror Descent for Reinforcement Learning: Linear Convergence, New Sampling Complexity, and Generalized Problem Classescited by 156
  4. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
  5. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  6. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 122
  7. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  8. 2019On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximationcited by 54
  9. 2019A Finite-Time Analysis of Q-Learning with Neural Network Function Approximationcited by 47
  10. 2021Finite Sample Analysis of Two-Time-Scale Natural Actor-Critic Algorithmcited by 39
  11. 2019Distributed Reinforcement Learning for Decentralized Linear Quadratic Control: A Derivative-Free Policy Optimization Approachcited by 38
  12. 2022Finite-time analysis of single-timescale actor-criticcited by 32
7 more
  1. 2020Momentum-Based Policy Gradient Methodscited by 30
  2. 2022A Small Gain Analysis of Single Timescale Actor Criticcited by 24
  3. 2019Policy Optimization with Stochastic Mirror Descentcited by 18
  4. 2022A Single-Timescale Analysis For Stochastic Approximation With Multiple Coupled Sequencescited by 17
  5. 2020Stochastic Linear Quadratic Optimal Control Problem: A Reinforcement Learning Methodcited by 11
  6. 2020Adaptive Temporal Difference Learning with Linear Function Approximationcited by 9
  7. 2021Robotic Knee Tracking Control to Mimic the Intact Human Knee Profile Based on Actor-critic Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Actor-Critics Can Achieve Optimal Sample Efficiency
  2. 2026A Pontryagin Method of Model-based Reinforcement Learning via Hamiltonian Actor-Critic
  3. 2026Heavy-Ball Q-Learning with Residual Weighting Correction
  4. 2025Finite-time Convergence Analysis of Actor-Critic with Evolving Reward
  5. 2024Compatible Gradient Approximations for Actor-Critic Algorithms
  6. 2024A Sharper Global Convergence Analysis for Average Reward Reinforcement Learning via an Actor-Critic Approach
  7. 2026Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum
  8. 2026Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition