# Proximal Policy Optimization with Relative Pearson Divergence

arXiv: https://arxiv.org/abs/2010.03290 · HTML: https://mlsanity.com/paper/2010.03290/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 9 of 9)

- [L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learning](https://mlsanity.com/paper/2202.07152/) (2022)
- [Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization](https://mlsanity.com/paper/2105.12991/) (2021)
- [Consolidated Adaptive T-soft Update for Deep Reinforcement Learning](https://mlsanity.com/paper/2202.12504/) (2022)
- [Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio](https://mlsanity.com/paper/2203.09809/) (2022)
- [Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures](https://mlsanity.com/paper/2104.00385/) (2021)
- [CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metric](https://mlsanity.com/paper/2110.10522/) (2021)
- [Design of Restricted Normalizing Flow towards Arbitrary Stochastic Policy with Computational Efficiency](https://mlsanity.com/paper/2412.12894/) (2024)
- [Fine-tuning Timeseries Predictors Using Reinforcement Learning](https://mlsanity.com/paper/2603.20063/) (2026)
- [Bounded Ratio Reinforcement Learning](https://mlsanity.com/paper/2604.18578/) (2026)

## Abstract (arXiv)

The recent remarkable progress of deep reinforcement learning (DRL) stands on regularization of policy for stable and efficient learning. A popular method, named proximal policy optimization (PPO), has been introduced for this purpose. PPO clips density ratio of the latest and baseline policies with a threshold, while its minimization target is unclear. As another problem of PPO, the symmetric threshold is given numerically while the density ratio itself is in asymmetric domain, thereby causing unbalanced regularization of the policy. This paper therefore proposes a new variant of PPO by considering a regularization problem of relative Pearson (RPE) divergence, so-called PPO-RPE. This regularization yields the clear minimization target, which constrains the latest policy to the baseline one. Through its analysis, the intuitive threshold-based design consistent with the asymmetry of the threshold and the domain of density ratio can be derived. Through four benchmark tasks, PPO-RPE performed as well as or better than the conventional methods in terms of the task performance by the learned policy.