mlsanityData to 13 Oct 2026

Characterizing the Gap Between Actor-Critic and Policy Gradient

arXiv 2106.06932 ↗First posted 13 Jun 2021cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Actor-critic (AC) methods are ubiquitous in reinforcement learning. Although it is understood that AC methods are closely related to policy gradient (PG), their precise connection has not been fully characterized previously. In this paper, we explain the gap between AC and PG methods by identifying the exact adjustment to the AC objective/gradient that recovers the true policy gradient of the cumulative reward objective (PG). Furthermore, by viewing the AC method as a two-player Stackelberg game between the actor and critic, we show that the Stackelberg policy gradient can be recovered as a special case of our more general analysis. Based on these results, we develop practical algorithms, Residual Actor-Critic and Stackelberg Actor-Critic, for estimating the correction between AC and PG and use these to modify the standard AC algorithm. Experiments on popular tabular and continuous environments show the proposed corrections can improve both the sample efficiency and final performance of existing AC methods.

Key facts. Cited by 9 papers in the map (data through 13 Oct 2026). Builds on 16 of its 38 references that are in the map. First posted on arXiv 13 Jun 2021.

Builds on

16 of 38 references.

References · 16

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  2. 2015Continuous control with deep reinforcement learningcited by 3,290
  3. 2015Trust Region Policy Optimizationcited by 2,513
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,310
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,940
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,185
  8. 2015Prioritized Experience Replaycited by 1,078
  9. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  10. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  11. 2017A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applicationscited by 127
  12. 2015Gradient Estimation Using Stochastic Computation Graphscited by 120
4 more
  1. 2020A Game Theoretic Framework for Model Based Reinforcement Learningcited by 72
  2. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  3. 2020An operator view of policy gradient methodscited by 17
  4. 2013Planning by Prioritized Sweeping with Small Backupscited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 2; 2022: 3; 2023: 1; 2024: 0; 2025: 2; 2026: 1.

9 papers.

  1. 2021Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithmscited by 18
  2. 2022Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approachcited by 14
  3. 2021Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximationcited by 13
  4. 2023Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspectivecited by 4
  5. 2025Bi-Level Policy Optimization with Nystr"om Hypergradientscited by 3
  6. 2022Reinforcement Learning for Personalized Drug Discovery and Design for Complex Diseases: A Systems Pharmacology Perspectivecited by 1
  7. 2022A Parametric Class of Approximate Gradient Updates for Policy Optimizationcited by 0
  8. 2025Subjective functionscited by 0
  9. 2026Verifying Neural Networks with Reinforcement Learningcited by 0