# Proximal Policy Gradient: PPO with Policy Gradient

arXiv: https://arxiv.org/abs/2010.09933 · HTML: https://mlsanity.com/paper/2010.09933/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping](https://mlsanity.com/paper/2312.12065/) (2023)
- [Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process](https://mlsanity.com/paper/2401.03244/) (2024)
- [Neural PPO-Clip Attains Global Optimality: A Hinge Loss Perspective](https://mlsanity.com/paper/2110.13799/) (2021)
- [DQNAS: Neural Architecture Search using Reinforcement Learning](https://mlsanity.com/paper/2301.06687/) (2023)
- [Recursive Least Squares Advantage Actor-Critic Algorithms](https://mlsanity.com/paper/2201.05918/) (2022)

## Abstract (arXiv)

In this paper, we propose a new algorithm PPG (Proximal Policy Gradient), which is close to both VPG (vanilla policy gradient) and PPO (proximal policy optimization). The PPG objective is a partial variation of the VPG objective and the gradient of the PPG objective is exactly same as the gradient of the VPG objective. To increase the number of policy update iterations, we introduce the advantage-policy plane and design a new clipping strategy. We perform experiments in OpenAI Gym and Bullet robotics environments for ten random seeds. The performance of PPG is comparable to PPO, and the entropy decays slower than PPG. Thus we show that performance similar to PPO can be obtained by using the gradient formula from the original policy gradient theorem.