# Sample-based Distributional Policy Gradient

arXiv: https://arxiv.org/abs/2001.02652 · HTML: https://mlsanity.com/paper/2001.02652/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Jan 2020.

## Built on by (most cited first, 14 of 14)

- [GMAC: A Distributional Perspective on Actor-Critic Framework](https://mlsanity.com/paper/2105.11366/) (2021)
- [Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning](https://mlsanity.com/paper/2309.14246/) (2023)
- [Unleashing Flow Policies with Distributional Critics](https://mlsanity.com/paper/2509.23087/) (2025)
- [Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control](https://mlsanity.com/paper/2207.13730/) (2022)
- [Sample-based Uncertainty Quantification with a Single Deterministic Neural Network](https://mlsanity.com/paper/2209.08418/) (2022)
- [Decision-making with Speculative Opponent Models](https://mlsanity.com/paper/2211.11940/) (2022)
- [PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm](https://mlsanity.com/paper/2306.06637/) (2023)
- [Distributional Reinforcement Learning based Integrated Decision Making and Control for Autonomous Surface Vehicles](https://mlsanity.com/paper/2412.09466/) (2024)
- [Normality-Guided Distributional Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2208.13125/) (2022)
- [Policy Evaluation in Distributional LQR](https://mlsanity.com/paper/2303.13657/) (2023)
- [Uncertainty-Aware Transient Stability-Constrained Preventive Redispatch: A Distributional Reinforcement Learning Approach](https://mlsanity.com/paper/2402.09263/) (2024)
- [On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2405.14749/) (2024)
- [Bag of Policies for Distributional Deep Exploration](https://mlsanity.com/paper/2308.01759/) (2023)
- [Distributional Active Inference](https://mlsanity.com/paper/2601.20985/) (2026)

## Abstract (arXiv)

Distributional reinforcement learning (DRL) is a recent reinforcement learning framework whose success has been supported by various empirical studies. It relies on the key idea of replacing the expected return with the return distribution, which captures the intrinsic randomness of the long term rewards. Most of the existing literature on DRL focuses on problems with discrete action space and value based methods. In this work, motivated by applications in robotics with continuous action space control settings, we propose sample-based distributional policy gradient (SDPG) algorithm. It models the return distribution using samples via a reparameterization technique widely used in generative modeling and inference. We compare SDPG with the state-of-art policy gradient method in DRL, distributed distributional deterministic policy gradients (D4PG), which has demonstrated state-of-art performance. We apply SDPG and D4PG to multiple OpenAI Gym environments and observe that our algorithm shows better sample efficiency as well as higher reward for most tasks.