# Natural Policy Gradients In Reinforcement Learning Explained

arXiv: https://arxiv.org/abs/2209.01820 · HTML: https://mlsanity.com/paper/2209.01820/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 8 references that are in the map.
- First posted on arXiv 5 Sep 2022.

## Builds on (most cited first, 2 of 2)

- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)

## Built on by (most cited first, 3 of 3)

- [Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applications](https://mlsanity.com/paper/2408.10215/) (2024)
- [Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning](https://mlsanity.com/paper/2601.18626/) (2026)
- [Efficient Hypergradient Descent for Inverse Reinforcement Learning](https://mlsanity.com/paper/2608.11052/) (2026)

## Abstract (arXiv)

Traditional policy gradient methods are fundamentally flawed. Natural gradients converge quicker and better, forming the foundation of contemporary Reinforcement Learning such as Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO). This lecture note aims to clarify the intuition behind natural policy gradients, focusing on the thought process and the key mathematical constructs.