# Optimistic Distributionally Robust Policy Optimization

arXiv: https://arxiv.org/abs/2006.07815 · HTML: https://mlsanity.com/paper/2006.07815/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2020.

## Built on by (most cited first, 6 of 6)

- [Differentiable Trust Region Layers for Deep Reinforcement Learning](https://mlsanity.com/paper/2101.09207/) (2021)
- [Maximum Entropy Reinforcement Learning with Mixture Policies](https://mlsanity.com/paper/2103.10176/) (2021)
- [A data-driven approach to beating SAA out-of-sample](https://mlsanity.com/paper/2105.12342/) (2021)
- [Rockafellian Relaxation and Stochastic Optimization under Perturbations](https://mlsanity.com/paper/2204.04762/) (2022)
- [On Tractability, Complexity, and Mixed-Integer Convex Programming Representability of Distributionally Favorable Optimization](https://mlsanity.com/paper/2401.17899/) (2024)
- [Breaking the Curse of Repulsion: Remoteness-Aware Control of Negative Off-Policy Updates](https://mlsanity.com/paper/2602.10430/) (2026)

## Abstract (arXiv)

Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), as the widely employed policy based reinforcement learning (RL) methods, are prone to converge to a sub-optimal solution as they limit the policy representation to a particular parametric distribution class. To address this issue, we develop an innovative Optimistic Distributionally Robust Policy Optimization (ODRPO) algorithm, which effectively utilizes Optimistic Distributionally Robust Optimization (DRO) approach to solve the trust region constrained optimization problem without parameterizing the policies. Our algorithm improves TRPO and PPO with a higher sample efficiency and a better performance of the final policy while attaining the learning stability. Moreover, it achieves a globally optimal policy update that is not promised in the prevailing policy based RL algorithms. Experiments across tabular domains and robotic locomotion tasks demonstrate the effectiveness of our approach.