# Stochastically Dominant Distributional Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.07318 · HTML: https://mlsanity.com/paper/1905.07318/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 May 2019.

## Built on by (most cited first, 12 of 12)

- [Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning](https://mlsanity.com/paper/2205.12184/) (2022)
- [Expected Scalarised Returns Dominance: A New Solution Concept for Multi-Objective Decision Making](https://mlsanity.com/paper/2106.01048/) (2021)
- [Monte Carlo Tree Search Algorithms for Risk-Aware and Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2211.13032/) (2022)
- [Utility-Based Reinforcement Learning: Unifying Single-objective and Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2402.02665/) (2024)
- [Bayesian Distributional Policy Gradients](https://mlsanity.com/paper/2103.11265/) (2021)
- [Learning with Stochastic Orders](https://mlsanity.com/paper/2205.13684/) (2022)
- [Distributional Multi-Objective Decision Making](https://mlsanity.com/paper/2305.05560/) (2023)
- [Distributional Reinforcement Learning with Dual Expectile-Quantile Regression](https://mlsanity.com/paper/2305.16877/) (2023)
- [Beyond Expectations: Learning with Stochastic Dominance Made Practical](https://mlsanity.com/paper/2402.02698/) (2024)
- [When is Cognitive Radar Beneficial?](https://mlsanity.com/paper/2212.00597/) (2022)
- [Bag of Policies for Distributional Deep Exploration](https://mlsanity.com/paper/2308.01759/) (2023)
- [Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning](https://mlsanity.com/paper/2408.02295/) (2024)

## Abstract (arXiv)

We describe a new approach for managing aleatoric uncertainty in the Reinforcement Learning (RL) paradigm. Instead of selecting actions according to a single statistic, we propose a distributional method based on the second-order stochastic dominance (SSD) relation. This compares the inherent dispersion of random returns induced by actions, producing a more comprehensive and robust evaluation of the environment's uncertainty. The necessary conditions for SSD require estimators to predict accurate second moments. To accommodate this, we map the distributional RL problem to a Wasserstein gradient flow, treating the distributional Bellman residual as a potential energy functional. We propose a particle-based algorithm for which we prove optimality and convergence. Our experiments characterize the algorithm performance and demonstrate how uncertainty and performance are better balanced using an textscssd policy than with other risk measures.