mlsanityData to 14 Oct 2026

Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds

arXiv 2210.14051 ↗First posted 25 Oct 2022cs.LG cs.AI stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

We study the regret guarantee for risk-sensitive reinforcement learning (RSRL) via distributional reinforcement learning (DRL) methods. In particular, we consider finite episodic Markov decision processes whose objective is the entropic risk measure (EntRM) of return. By leveraging a key property of the EntRM, the independence property, we establish the risk-sensitive distributional dynamic programming framework. We then propose two novel DRL algorithms that implement optimism through two different schemes, including a model-free one and a model-based one. We prove that they both attain tildeO(fracexp(|β| H)-1|β|H√S2AK) regret upper bound, where S, A, K, and H represent the number of states, actions, episodes, and the time horizon, respectively. It matches RSVI2 proposed in citefei2021exponential, with novel distributional analysis. To the best of our knowledge, this is the first regret analysis that bridges DRL and RSRL in terms of sample complexity. Acknowledging the computational inefficiency associated with the model-free DRL algorithm, we propose an alternative DRL algorithm with distribution representation. This approach not only maintains the established regret bounds but also significantly amplifies computational efficiency. We also prove a tighter minimax lower bound of Omega(fracexp(β H/6)-1β HH√SAT) for the β>0 case, which recovers the tight lower bound Omega(H√SAT) in the risk-neutral setting.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 18 of its 42 references that are in the map. First posted on arXiv 25 Oct 2022.

Builds on

18 of 42 references.

References · 18

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  3. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  4. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  5. 2013Risk-sensitive Reinforcement Learningcited by 130
  6. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 106
  7. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  8. 2016Explore First, Exploit Next: The True Shape of Regret in Bandit Problemscited by 79
  9. 2021Conservative Offline Distributional Reinforcement Learningcited by 79
  10. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  11. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  12. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
6 more
  1. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  2. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  3. 2020Improving Robustness via Risk Averse Distributional Reinforcement Learningcited by 28
  4. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 22
  5. 2019Entropic Risk Measure in Policy Searchcited by 19
  6. 2021Robustness and risk management via distributional dynamic programmingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 5; 2025: 3; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  2. 2023Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalentscited by 15
  3. 2023Beyond Average Return in Markov Decision Processescited by 11
  4. 2023Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measurescited by 8
  5. 2023Risk-sensitive Markov Decision Process and Learning under General Utility Functionscited by 7
  6. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Modelcited by 7
  7. 2025Efficient Risk-sensitive Planning via Entropic Risk Measurescited by 7
  8. 2024Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgentcited by 6
  9. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learningcited by 6
  10. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learningcited by 6
  11. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
  12. 2025Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and λ-Effectivenesscited by 2
4 more
  1. 2023Policy Evaluation in Distributional LQRcited by 1
  2. 2024Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximationcited by 0
  3. 2026Pessimistic Risk-Aware Policy Learning in Contextual Banditscited by 0
  4. 2026Learning Chance-Constrained MDPs with Bellman Distributional Certificatescited by 0