# Compatible Natural Gradient Policy Search

arXiv: https://arxiv.org/abs/1902.02823 · HTML: https://mlsanity.com/paper/1902.02823/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2019.

## Built on by (most cited first, 10 of 10)

- [Re-parameterizing Your Optimizers rather than Architectures](https://mlsanity.com/paper/2205.15242/) (2022)
- [Differentiable Trust Region Layers for Deep Reinforcement Learning](https://mlsanity.com/paper/2101.09207/) (2021)
- [Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inference](https://mlsanity.com/paper/2508.12511/) (2025)
- [A Unified Perspective on Natural Gradient Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/2209.11533/) (2022)
- [Learning Boltzmann Generators via Constrained Mass Transport](https://mlsanity.com/paper/2510.18460/) (2025)
- [Relative Entropy Pathwise Policy Optimization](https://mlsanity.com/paper/2507.11019/) (2025)
- [Towards Principled Unsupervised Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2502.08365/) (2025)
- [Absolute Policy Optimization](https://mlsanity.com/paper/2310.13230/) (2023)
- [Regret-Aware Black-Box Optimization with Natural Gradients, Trust-Regions and Entropy Control](https://mlsanity.com/paper/2206.06090/) (2022)
- [Differentially Private Natural Gradient Descent](https://mlsanity.com/paper/2607.05866/) (2026)

## Abstract (arXiv)

Trust-region methods have yielded state-of-the-art results in policy search. A common approach is to use KL-divergence to bound the region of trust resulting in a natural gradient policy update. We show that the natural gradient and trust region optimization are equivalent if we use the natural parameterization of a standard exponential policy distribution in combination with compatible value function approximation. Moreover, we show that standard natural gradient updates may reduce the entropy of the policy according to a wrong schedule leading to premature convergence. To control entropy reduction we introduce a new policy search method called compatible policy search (COPOS) which bounds entropy loss. The experimental results show that COPOS yields state-of-the-art results in challenging continuous control tasks and in discrete partially observable tasks.