# Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds

arXiv: https://arxiv.org/abs/2210.14051 · HTML: https://mlsanity.com/paper/2210.14051/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 25 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Distributed Distributional Deterministic Policy Gradients](https://mlsanity.com/paper/1804.08617/) (2018)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisited](https://mlsanity.com/paper/2010.03531/) (2020)
- [Fully Parameterized Quantile Function for Distributional Reinforcement Learning](https://mlsanity.com/paper/1911.02140/) (2019)
- [Explore First, Exploit Next: The True Shape of Regret in Bandit Problems](https://mlsanity.com/paper/1602.07182/) (2016)
- [Conservative Offline Distributional Reinforcement Learning](https://mlsanity.com/paper/2107.06106/) (2021)
- [An Analysis of Categorical Distributional Reinforcement Learning](https://mlsanity.com/paper/1802.08163/) (2018)
- [Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy](https://mlsanity.com/paper/1911.01546/) (2019)
- [Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret](https://mlsanity.com/paper/2006.13827/) (2020)
- [A Comparative Analysis of Expected and Distributional Reinforcement Learning](https://mlsanity.com/paper/1901.11084/) (2019)
- [Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2111.03947/) (2021)
- [Improving Robustness via Risk Averse Distributional Reinforcement Learning](https://mlsanity.com/paper/2005.00585/) (2020)
- [Distributional Reinforcement Learning for Multi-Dimensional Reward Functions](https://mlsanity.com/paper/2110.13578/) (2021)
- [Entropic Risk Measure in Policy Search](https://mlsanity.com/paper/1906.09090/) (2019)
- [Robustness and risk management via distributional dynamic programming](https://mlsanity.com/paper/2112.15430/) (2021)

## Built on by (most cited first, 16 of 16)

- [Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaR](https://mlsanity.com/paper/2302.03201/) (2023)
- [Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents](https://mlsanity.com/paper/2301.12601/) (2023)
- [Beyond Average Return in Markov Decision Processes](https://mlsanity.com/paper/2310.20266/) (2023)
- [Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures](https://mlsanity.com/paper/2306.02399/) (2023)
- [Risk-sensitive Markov Decision Process and Learning under General Utility Functions](https://mlsanity.com/paper/2311.13589/) (2023)
- [Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model](https://mlsanity.com/paper/2402.07598/) (2024)
- [Efficient Risk-sensitive Planning via Entropic Risk Measures](https://mlsanity.com/paper/2502.20423/) (2025)
- [Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent](https://mlsanity.com/paper/2402.10228/) (2024)
- [Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning](https://mlsanity.com/paper/2407.07631/) (2024)
- [Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning](https://mlsanity.com/paper/2501.02087/) (2025)
- [Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation](https://mlsanity.com/paper/2402.18159/) (2024)
- [Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and λ-Effectiveness](https://mlsanity.com/paper/2511.07604/) (2025)
- [Policy Evaluation in Distributional LQR](https://mlsanity.com/paper/2303.13657/) (2023)
- [Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation](https://mlsanity.com/paper/2407.21260/) (2024)
- [Pessimistic Risk-Aware Policy Learning in Contextual Bandits](https://mlsanity.com/paper/2605.15620/) (2026)
- [Learning Chance-Constrained MDPs with Bellman Distributional Certificates](https://mlsanity.com/paper/2609.30856/) (2026)

## Abstract (arXiv)

We study the regret guarantee for risk-sensitive reinforcement learning (RSRL) via distributional reinforcement learning (DRL) methods. In particular, we consider finite episodic Markov decision processes whose objective is the entropic risk measure (EntRM) of return. By leveraging a key property of the EntRM, the independence property, we establish the risk-sensitive distributional dynamic programming framework. We then propose two novel DRL algorithms that implement optimism through two different schemes, including a model-free one and a model-based one. We prove that they both attain tildeO(fracexp(|β| H)-1|β|H√S2AK) regret upper bound, where S, A, K, and H represent the number of states, actions, episodes, and the time horizon, respectively. It matches RSVI2 proposed in citefei2021exponential, with novel distributional analysis. To the best of our knowledge, this is the first regret analysis that bridges DRL and RSRL in terms of sample complexity. Acknowledging the computational inefficiency associated with the model-free DRL algorithm, we propose an alternative DRL algorithm with distribution representation. This approach not only maintains the established regret bounds but also significantly amplifies computational efficiency. We also prove a tighter minimax lower bound of Omega(fracexp(β H/6)-1β HH√SAT) for the β>0 case, which recovers the tight lower bound Omega(H√SAT) in the risk-neutral setting.