# Robust Policy Optimization in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2212.07536 · HTML: https://mlsanity.com/paper/2212.07536/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 14 Dec 2022.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [On the Global Convergence Rates of Softmax Policy Gradient Methods](https://mlsanity.com/paper/2005.06392/) (2020)
- [The Primacy Bias in Deep Reinforcement Learning](https://mlsanity.com/paper/2205.07802/) (2022)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [Automatic Data Augmentation for Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/2006.12862/) (2020)
- [Bootstrap State Representation using Style Transfer for Better Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/2207.07749/) (2022)

## Built on by (most cited first, 15 of 15)

- [Real-Time Execution of Action Chunking Flow Policies](https://mlsanity.com/paper/2506.07339/) (2025)
- [Parseval Regularization for Continual Reinforcement Learning](https://mlsanity.com/paper/2412.07224/) (2024)
- [VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimization](https://mlsanity.com/paper/2512.16906/) (2025)
- [The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations](https://mlsanity.com/paper/2401.13662/) (2024)
- [Learning Adaptive Safety for Multi-Agent Systems](https://mlsanity.com/paper/2309.10657/) (2023)
- [Accelerating Policy Gradient by Estimating Value Function from Prior Computation in Deep Reinforcement Learning](https://mlsanity.com/paper/2302.01399/) (2023)
- [KIPPO: Koopman-Inspired Proximal Policy Optimization](https://mlsanity.com/paper/2505.14566/) (2025)
- [Navigation of micro-robot swarms for targeted delivery using reinforcement learning](https://mlsanity.com/paper/2306.17598/) (2023)
- [Learning with Expert Abstractions for Efficient Multi-Task Continuous Control](https://mlsanity.com/paper/2503.14809/) (2025)
- [Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Critic](https://mlsanity.com/paper/2506.05445/) (2025)
- [An effective control of large systems of active particles: An application to evacuation problem](https://mlsanity.com/paper/2509.19972/) (2025)
- [Rich State Observations Empower Reinforcement Learning to Surpass PID: A Drone Ball Balancing Study](https://mlsanity.com/paper/2509.21122/) (2025)
- [Design for Manufacturing: A Manufacturability Knowledge-Integrated Reinforcement Learning Framework for Free-Form Pipe Routing in Aeroengines](https://mlsanity.com/paper/2605.20644/) (2026)
- [What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor](https://mlsanity.com/paper/2609.32532/) (2026)
- [Temporally Interpretable Differentiable Decision Trees](https://mlsanity.com/paper/2610.10367/) (2026)

## Abstract (arXiv)

The policy gradient method enjoys the simplicity of the objective where the agent optimizes the cumulative reward directly. Moreover, in the continuous action domain, parameterized distribution of action distribution allows easy control of exploration, resulting from the variance of the representing distribution. Entropy can play an essential role in policy optimization by selecting the stochastic policy, which eventually helps better explore the environment in reinforcement learning (RL). However, the stochasticity often reduces as the training progresses; thus, the policy becomes less exploratory. Additionally, certain parametric distributions might only work for some environments and require extensive hyperparameter tuning. This paper aims to mitigate these issues. In particular, we propose an algorithm called Robust Policy Optimization (RPO), which leverages a perturbed distribution. We hypothesize that our method encourages high-entropy actions and provides a way to represent the action space better. We further provide empirical evidence to verify our hypothesis. We evaluated our methods on various continuous control tasks from DeepMind Control, OpenAI Gym, Pybullet, and IsaacGym. We observed that in many settings, RPO increases the policy entropy early in training and then maintains a certain level of entropy throughout the training period. Eventually, our agent RPO shows consistently improved performance compared to PPO and other techniques: entropy regularization, different distributions, and data augmentation. Furthermore, in several settings, our method stays robust in performance, while other baseline mechanisms fail to improve and even worsen the performance.