# A2C is a special case of PPO

arXiv: https://arxiv.org/abs/2205.09123 · HTML: https://mlsanity.com/paper/2205.09123/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 18 May 2022.

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [ChainerRL: A Deep Reinforcement Learning Library](https://mlsanity.com/paper/1912.03905/) (2019)
- [MushroomRL: Simplifying Reinforcement Learning Research](https://mlsanity.com/paper/2001.01102/) (2020)
- [Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games](https://mlsanity.com/paper/2007.01542/) (2020)

## Built on by (most cited first, 16 of 16)

- [Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models](https://mlsanity.com/paper/2410.18252/) (2024)
- [What makes math problems hard for reinforcement learning: a case study](https://mlsanity.com/paper/2408.15332/) (2024)
- [Thinking Preference Optimization](https://mlsanity.com/paper/2502.13173/) (2025)
- [Quantum Architecture Search with Unsupervised Representation Learning](https://mlsanity.com/paper/2401.11576/) (2024)
- [Monte Carlo Beam Search for Actor-Critic Reinforcement Learning in Continuous Control](https://mlsanity.com/paper/2505.09029/) (2025)
- [Target-independent XLA optimization using Reinforcement Learning](https://mlsanity.com/paper/2308.14364/) (2023)
- [New Adaptive Mechanism for Large Neighborhood Search using Dual Actor-Critic](https://mlsanity.com/paper/2601.11414/) (2026)
- [Learning to Abstract Visuomotor Mappings using Meta-Reinforcement Learning](https://mlsanity.com/paper/2402.03072/) (2024)
- [Artificial Intelligence for Multi-Unit Auction design](https://mlsanity.com/paper/2404.15633/) (2024)
- [A Simple, Solid, and Reproducible Baseline for Bridge Bidding AI](https://mlsanity.com/paper/2406.10306/) (2024)
- [Optimizing 2D+1 Packing in Constrained Environments Using Deep Reinforcement Learning](https://mlsanity.com/paper/2503.17573/) (2025)
- [Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.18433/) (2025)
- [Failure Modes of Maximum Entropy RLHF](https://mlsanity.com/paper/2509.20265/) (2025)
- [Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference](https://mlsanity.com/paper/2510.21184/) (2025)
- [Taxonomy and Trends in Reinforcement Learning for Robotics and Control Systems: A Structured Review](https://mlsanity.com/paper/2510.21758/) (2025)
- [Trading Confidence: Comprehensive Uncertainty Estimation in Algorithmic Trading](https://mlsanity.com/paper/2607.02864/) (2026)

## Abstract (arXiv)

Advantage Actor-critic (A2C) and Proximal Policy Optimization (PPO) are popular deep reinforcement learning algorithms used for game AI in recent years. A common understanding is that A2C and PPO are separate algorithms because PPO's clipped objective appears significantly different than A2C's objective. In this paper, however, we show A2C is a special case of PPO. We present theoretical justifications and pseudocode analysis to demonstrate why. To validate our claim, we conduct an empirical experiment using textttStable-baselines3, showing A2C and PPO produce the textitexact same models when other settings are controlled.