Markov Decision Processes of the Third Kind: Learning Distributions by Policy Gradient Descent
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
The goal of this paper is to analyze distributional Markov Decision Processes as a class of control problems in which the objective is to learn policies that steer the distribution of a cumulative reward toward a prescribed target law, rather than optimizing an expected value or a risk functional. To solve the resulting distributional control problem in a model-free setting, we propose a policy-gradient algorithm based on neural-network parameterizations of randomized Markov policies, defined on an augmented state space and a sample-based evaluation of the characteristic-function loss. Under mild regularity and growth assumptions, we prove convergence of the algorithm to stationary points using stochastic approximation techniques. Several numerical experiments illustrate the ability of the method to match complex target distributions, recover classical optimal policies when they exist, and reveal intrinsic non-uniqueness phenomena specific to distributional control.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 40 references that are in the map. First posted on arXiv 6 Feb 2026.
Builds on
18 of 40 references.
Closest ancestors · 4
Same direction, newest first
- 2025Tractable Representations for Convergent Approximation of Distributional HJB Equations7 Mar
- 2025An Actor-Critic Algorithm with Function Approximation for Risk Sensitive Cost Markov Decision Processes17 Feb
- 2025Optimizing Return Distributions with Distributional Dynamic Programming22 Jan
- 2024Robust Reinforcement Learning with Dynamic Distortion Risk Measures16 Sep
Other references · 14
- 2017A Distributional Perspective on Reinforcement Learningcited by 765
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
- 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
- 2016Deep Learning Approximation for Stochastic Control Problemscited by 55
- 2023High Probability Convergence of Stochastic Gradient Methodscited by 49
- 2018Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysiscited by 24
- 2021Reinforcement Learning with Dynamic Convex Risk Measurescited by 21
- 2019Distributional reinforcement learning with linear function approximationcited by 16
- 2020Distributionally Robust Markov Decision Processes and their Connection to Risk Measurescited by 14
- 2022Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learningcited by 12
- 2025Robust time series generation via Schr"odinger Bridge: a comprehensive evaluationcited by 8
- 2023Dynamic Programming in Probability Spaces via Optimal Transportcited by 6
2 more
- 2023One-Step Distributional Reinforcement Learningcited by 6
- 2025Yet Another Distributional Bellman Equationcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024On Policy Evaluation Algorithms in Distributional Reinforcement Learning
- 2026Policy Gradient Methods for Non-Markovian Reinforcement Learning
- 2024Control randomisation approach for policy gradient and application to reinforcement learning in optimal switching
- 2024Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPs
- 2024On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes
- 2025A Differential Perspective on Distributional Reinforcement Learning
- 2025Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions
- 2026Policy Gradient Learning for Distributionally Robust Markov Decision Processes under Wasserstein Ambiguity