# Distributional Policy Optimization: An Alternative Approach for Continuous Control

arXiv: https://arxiv.org/abs/1905.09855 · HTML: https://mlsanity.com/paper/1905.09855/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2019.

## Built on by (most cited first, 22 of 22)

- [Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning](https://mlsanity.com/paper/2310.11731/) (2023)
- [Distributional Soft Actor-Critic with Three Refinements](https://mlsanity.com/paper/2310.05858/) (2023)
- [Continuous Control with Action Quantization from Demonstrations](https://mlsanity.com/paper/2110.10149/) (2021)
- [Reinforcement Learning with History-Dependent Dynamic Contexts](https://mlsanity.com/paper/2302.02061/) (2023)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Optimistic Curiosity Exploration and Conservative Exploitation with Linear Reward Shaping](https://mlsanity.com/paper/2209.07288/) (2022)
- [Provably Convergent Policy Optimization via Metric-aware Trust Region Methods](https://mlsanity.com/paper/2306.14133/) (2023)
- [Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusion](https://mlsanity.com/paper/2405.19528/) (2024)
- [Maximum Entropy Reinforcement Learning with Mixture Policies](https://mlsanity.com/paper/2103.10176/) (2021)
- [Representation-Driven Reinforcement Learning](https://mlsanity.com/paper/2305.19922/) (2023)
- [Decision-making with Speculative Opponent Models](https://mlsanity.com/paper/2211.11940/) (2022)
- [Coordinate Ascent for Off-Policy RL with Global Convergence Guarantees](https://mlsanity.com/paper/2212.05237/) (2022)
- [Distillation Policy Optimization](https://mlsanity.com/paper/2302.00533/) (2023)
- [Latent-Conditioned Policy Gradient for Multi-Objective Deep Reinforcement Learning](https://mlsanity.com/paper/2303.08909/) (2023)
- [A Spectral Revisit of the Distributional Bellman Operator under the Cram'er Metric](https://mlsanity.com/paper/2603.12576/) (2026)
- [Towards Optimal Pricing of Demand Response -- A Nonparametric Constrained Policy Optimization Approach](https://mlsanity.com/paper/2306.14047/) (2023)
- [Learning in complex action spaces without policy gradients](https://mlsanity.com/paper/2410.06317/) (2024)
- [Adaptive Nesterov Accelerated Distributional Deep Hedging for Efficient Volatility Risk Management](https://mlsanity.com/paper/2502.17777/) (2025)
- [The DeepXube Software Package for Solving Pathfinding Problems with Learned Heuristic Functions and Search](https://mlsanity.com/paper/2603.23873/) (2026)
- [Distributional Value Estimation Without Target Networks for Robust Quality-Diversity](https://mlsanity.com/paper/2604.20381/) (2026)
- [Distributional Soft Bellman Operator under the Cram'er Geometry](https://mlsanity.com/paper/2607.17897/) (2026)
- [Value-Preserving Action Quantization](https://mlsanity.com/paper/2610.17252/) (2026)

## Abstract (arXiv)

We identify a fundamental problem in policy gradient-based methods in continuous control. As policy gradient methods require the agent's underlying probability distribution, they limit policy representation to parametric distribution classes. We show that optimizing over such sets results in local movement in the action space and thus convergence to sub-optimal solutions. We suggest a novel distributional framework, able to represent arbitrary distribution functions over the continuous action space. Using this framework, we construct a generative scheme, trained using an off-policy actor-critic paradigm, which we call the Generative Actor Critic (GAC). Compared to policy gradient methods, GAC does not require knowledge of the underlying probability distribution, thereby overcoming these limitations. Empirical evaluation shows that our approach is comparable and often surpasses current state-of-the-art baselines in continuous domains.