mlsanityData to 14 Oct 2026

Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Path

arXiv 2206.02678 ↗First posted 6 Jun 2022cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

In this paper, we study a novel episodic risk-sensitive Reinforcement Learning (RL) problem, named Iterated CVaR RL, which aims to maximize the tail of the reward-to-go at each step, and focuses on tightly controlling the risk of getting into catastrophic situations at each stage. This formulation is applicable to real-world tasks that demand strong risk avoidance throughout the decision process, such as autonomous driving, clinical treatment planning and robotics. We investigate two performance metrics under Iterated CVaR RL, i.e., Regret Minimization and Best Policy Identification. For both metrics, we design efficient algorithms ICVaR-RM and ICVaR-BPI, respectively, and provide nearly matching upper and lower bounds with respect to the number of episodes K. We also investigate an interesting limiting case of Iterated CVaR RL, called Worst Path RL, where the objective becomes to maximize the minimum possible cumulative reward. For Worst Path RL, we propose an efficient algorithm with constant upper and lower bounds. Finally, our techniques for bounding the change of CVaR due to the value function shift and decomposing the regret via a distorted visitation distribution are novel, and can find applications in other risk-sensitive RL problems.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 18 of its 52 references that are in the map. First posted on arXiv 6 Jun 2022.

Builds on

18 of 52 references.

References · 18

  1. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  2. 2018Is Q-learning Provably Efficient?cited by 487
  3. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  4. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  5. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 195
  6. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 176
  7. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  8. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 106
  9. 2014Optimizing the CVaR via Samplingcited by 99
  10. 2014Algorithms for CVaR Optimization in MDPscited by 99
  11. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  12. 2020Adaptive Reward-Free Explorationcited by 77
6 more
  1. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  2. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
  3. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  4. 2020Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimizationcited by 35
  5. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  6. 2020Markov Decision Processes with Recursive Risk Measurescited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 6; 2026: 6.

22 papers.

Sort:

By area:

  1. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 16
  2. 2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedbackcited by 14
  3. 2023Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measurescited by 8
  4. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Modelcited by 7
  5. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimizationcited by 7
  6. 2023Provably Efficient CVaR RL in Low-rank MDPscited by 6
  7. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learningcited by 6
  8. 2025Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Modelcited by 5
  9. 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Modelcited by 5
  10. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
  11. 2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3
  12. 2025Risk-sensitive Reinforcement Learning Based on Convex Scoring Functionscited by 2
10 more
  1. 2025Risk-aware Direct Preference Optimization under Nested Risk Measurecited by 2
  2. 2026Online Risk-Averse Planning in POMDPs Using Iterated CVaR Value Functioncited by 2
  3. 2026On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalentscited by 1
  4. 2024Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observationcited by 0
  5. 2025Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inferencecited by 0
  6. 2025Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignmentcited by 0
  7. 2026Instantiating Bayesian CVaR lower bounds in Interactive Decision Making Problemscited by 0
  8. 2026Risk-Averse Online POMDP Planning via CVaR of the Immediate Cost with Performance Guaranteescited by 0
  9. 2026Robust Risk-Sensitive Reinforcement Learning from Corrupted Human Feedbackcited by 0
  10. 2026Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Modelcited by 0