# PPO-UE: Proximal Policy Optimization via Uncertainty-Aware Exploration

arXiv: https://arxiv.org/abs/2212.06343 · HTML: https://mlsanity.com/paper/2212.06343/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 13 Dec 2022.

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Better Exploration with Optimistic Actor-Critic](https://mlsanity.com/paper/1910.12807/) (2019)
- [PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation](https://mlsanity.com/paper/1810.02541/) (2018)
- [Proximal Policy Optimization via Enhanced Exploration Efficiency](https://mlsanity.com/paper/2011.05525/) (2020)

## Built on by (most cited first, 1 of 1)

- [Adaptive Vehicle Speed Classification via BMCNN with Reinforcement Learning-Enhanced Acoustic Processing](https://mlsanity.com/paper/2509.00839/) (2025)

## Abstract (arXiv)

Proximal Policy Optimization (PPO) is a highly popular policy-based deep reinforcement learning (DRL) approach. However, we observe that the homogeneous exploration process in PPO could cause an unexpected stability issue in the training phase. To address this issue, we propose PPO-UE, a PPO variant equipped with self-adaptive uncertainty-aware explorations (UEs) based on a ratio uncertainty level. The proposed PPO-UE is designed to improve convergence speed and performance with an optimized ratio uncertainty level. Through extensive sensitivity analysis by varying the ratio uncertainty level, our proposed PPO-UE considerably outperforms the baseline PPO in Roboschool continuous control tasks.