mlsanityData to 14 Oct 2026

A2C is a special case of PPO

arXiv 2205.09123 ↗First posted 18 May 2022cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Advantage Actor-critic (A2C) and Proximal Policy Optimization (PPO) are popular deep reinforcement learning algorithms used for game AI in recent years. A common understanding is that A2C and PPO are separate algorithms because PPO's clipped objective appears significantly different than A2C's objective. In this paper, however, we show A2C is a special case of PPO. We present theoretical justifications and pseudocode analysis to demonstrate why. To validate our claim, we conduct an empirical experiment using textttStable-baselines3, showing A2C and PPO produce the textitexact same models when other settings are controlled.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 8 of its 19 references that are in the map. First posted on arXiv 18 May 2022.

Builds on

8 of 19 references.

References · 8

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  6. 2019ChainerRL: A Deep Reinforcement Learning Librarycited by 51
  7. 2020MushroomRL: Simplifying Reinforcement Learning Researchcited by 50
  8. 2020Strategies for Using Proximal Policy Optimization in Mobile Puzzle Gamescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 7; 2026: 2.

16 papers.

Sort:

By area:

  1. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  2. 2024What makes math problems hard for reinforcement learning: a case studycited by 9
  3. 2025Thinking Preference Optimizationcited by 8
  4. 2024Quantum Architecture Search with Unsupervised Representation Learningcited by 4
  5. 2025Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Controlcited by 3
  6. 2023Target-independent XLA optimization using Reinforcement Learningcited by 1
  7. 2026New Adaptive Mechanism for Large Neighborhood Search using Dual Actor-Criticcited by 1
  8. 2024Learning to Abstract Visuomotor Mappings using Meta-Reinforcement Learningcited by 0
  9. 2024Artificial Intelligence for Multi-Unit Auction designcited by 0
  10. 2024A Simple, Solid, and Reproducible Baseline for Bridge Bidding AIcited by 0
  11. 2025Optimizing 2D+1 Packing in Constrained Environments Using Deep Reinforcement Learningcited by 0
  12. 2025Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learningcited by 0
4 more
  1. 2025Failure Modes of Maximum Entropy RLHFcited by 0
  2. 2025Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inferencecited by 0
  3. 2025Taxonomy and Trends in Reinforcement Learning for Robotics and Control Systems: A Structured Reviewcited by 0
  4. 2026Trading Confidence: Comprehensive Uncertainty Estimation in Algorithmic Tradingcited by 0