# PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation

arXiv: https://arxiv.org/abs/1810.02541 · HTML: https://mlsanity.com/paper/1810.02541/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2018.

## Built on by (most cited first, 14 of 14)

- [Kolmogorov-Arnold Network for Online Reinforcement Learning](https://mlsanity.com/paper/2408.04841/) (2024)
- [Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratio](https://mlsanity.com/paper/2203.09809/) (2022)
- [Autonomous Drone Swarm Navigation and Multi-target Tracking in 3D Environments with Dynamic Obstacles](https://mlsanity.com/paper/2202.06253/) (2022)
- [Optimistic Multi-Agent Policy Gradient](https://mlsanity.com/paper/2311.01953/) (2023)
- [Evolution Guided Generative Flow Networks](https://mlsanity.com/paper/2402.02186/) (2024)
- [Model-based deep reinforcement learning for accelerated learning from flow simulations](https://mlsanity.com/paper/2402.16543/) (2024)
- [Path Signatures for Diversity in Probabilistic Trajectory Optimisation](https://mlsanity.com/paper/2308.04071/) (2023)
- [Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning](https://mlsanity.com/paper/2605.00347/) (2026)
- [PPO-UE: Proximal Policy Optimization via Uncertainty-Aware Exploration](https://mlsanity.com/paper/2212.06343/) (2022)
- [RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning](https://mlsanity.com/paper/2601.23075/) (2026)
- [Self-Imitation Learning of Locomotion Movements through Termination Curriculum](https://mlsanity.com/paper/1907.11842/) (2019)
- [Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPO](https://mlsanity.com/paper/2608.02031/) (2026)
- [Learning High-Risk High-Precision Motion Control](https://mlsanity.com/paper/2609.34851/) (2026)
- [DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation](https://mlsanity.com/paper/2610.00360/) (2026)

## Abstract (arXiv)

Proximal Policy Optimization (PPO) is a highly popular model-free reinforcement learning (RL) approach. However, we observe that in a continuous action space, PPO can prematurely shrink the exploration variance, which leads to slow progress and may make the algorithm prone to getting stuck in local optima. Drawing inspiration from CMA-ES, a black-box evolutionary optimization method designed for robustness in similar situations, we propose PPO-CMA, a proximal policy optimization approach that adaptively expands the exploration variance to speed up progress. With only minor changes to PPO, our algorithm considerably improves performance in Roboschool continuous control benchmarks. Our results also show that PPO-CMA, as opposed to PPO, is significantly less sensitive to the choice of hyperparameters, allowing one to use it in complex movement optimization tasks without requiring tedious tuning.