# On Policy Gradients

arXiv: https://arxiv.org/abs/1911.04817 · HTML: https://mlsanity.com/paper/1911.04817/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2019.

## Built on by (most cited first, 6 of 6)

- [PLUTO: Pushing the Limit of Imitation Learning-based Planning for Autonomous Driving](https://mlsanity.com/paper/2404.14327/) (2024)
- [Diffusion Policies creating a Trust Region for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.19690/) (2024)
- [ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency](https://mlsanity.com/paper/2211.16068/) (2022)
- [An Analysis of Multi-Agent Reinforcement Learning for Decentralized Inventory Control Systems](https://mlsanity.com/paper/2307.11432/) (2023)
- [What If: Generating Code to Answer Simulation Questions](https://mlsanity.com/paper/2204.07835/) (2022)
- [Equivalence of stochastic and deterministic policy gradients](https://mlsanity.com/paper/2505.23244/) (2025)

## Abstract (arXiv)

The goal of policy gradient approaches is to find a policy in a given class of policies which maximizes the expected return. Given a differentiable model of the policy, we want to apply a gradient-ascent technique to reach a local optimum. We mainly use gradient ascent, because it is theoretically well researched. The main issue is that the policy gradient with respect to the expected return is not available, thus we need to estimate it. As policy gradient algorithms also tend to require on-policy data for the gradient estimate, their biggest weakness is sample efficiency. For this reason, most research is focused on finding algorithms with improved sample efficiency. This paper provides a formal introduction to policy gradient that shows the development of policy gradient approaches, and should enable the reader to follow current research on the topic.