mlsanityData to 7 Oct 2026

Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation

arXiv 2402.18159 ↗First posted 28 Feb 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In the realm of reinforcement learning (RL), accounting for risk is crucial for making decisions under uncertainty, particularly in applications where safety and reliability are paramount. In this paper, we introduce a general framework on Risk-Sensitive Distributional Reinforcement Learning (RS-DisRL), with static Lipschitz Risk Measures (LRM) and general function approximation. Our framework covers a broad class of risk-sensitive RL, and facilitates analysis of the impact of estimation functions on the effectiveness of RSRL strategies and evaluation of their sample complexity. We design two innovative meta-algorithms: textttRS-DisRL-M, a model-based strategy for model-based function approximation, and textttRS-DisRL-V, a model-free approach for general value function approximation. With our novel estimation techniques via Least Squares Regression (LSR) and Maximum Likelihood Estimation (MLE) in distributional RL with augmented Markov Decision Process (MDP), we derive the first widetildeO(√K) dependency of the regret upper bound for RSRL with static LRM, marking a pioneering contribution towards statistically efficient algorithms in this domain.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

34 of 55 references.

References · 34

  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  4. 2018Is Q-learning Provably Efficient?cited by 485
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  7. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  8. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  9. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  10. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  11. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  12. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
22 more
  1. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  2. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  3. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  4. 2018Offline Multi-Action Policy Learning: Generalization and Optimizationcited by 64
  5. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  6. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 57
  7. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
  8. 2022VOQL: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximationcited by 44
  9. 2021Towards General Function Approximation in Zero-Sum Markov Gamescited by 42
  10. 2022Doubly Robust Distributionally Robust Off-Policy Evaluation and Learningcited by 34
  11. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  12. 2022A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcited by 31
  13. 2023An Analysis of Quantile Temporal-Difference Learningcited by 30
  14. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  15. 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
  16. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
  17. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  18. 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
  19. 2023Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalentscited by 14
  20. 2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedbackcited by 13
  21. 2023Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measurescited by 7
  22. 2023Provably Efficient CVaR RL in Low-rank MDPscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learningcited by 6
  2. 2024Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximationcited by 0
  3. 2026Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learningcited by 0