mlsanityData to 7 Oct 2026

Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret

arXiv 2006.13827 ↗First posted 22 Jun 2020cs.LG math.OC stat.MLcited by 57

Not on the map. Why?

Abstract (arXiv)

We study risk-sensitive reinforcement learning in episodic Markov decision processes with unknown transition kernels, where the goal is to optimize the total reward under the risk measure of exponential utility. We propose two provably efficient model-free algorithms, Risk-Sensitive Value Iteration (RSVI) and Risk-Sensitive Q-learning (RSQ). These algorithms implement a form of risk-sensitive optimism in the face of uncertainty, which adapts to both risk-seeking and risk-averse modes of exploration. We prove that RSVI attains an tildeObig(λ(|β| H2) · √H3 S2AT big) regret, while RSQ attains an tildeObig(λ(|β| H2) · √H4 SAT big) regret, where λ(u) = (e3u-1)/u for u>0. In the above, β is the risk parameter of the exponential utility function, S the number of states, A the number of actions, T the total number of timesteps, and H the episode length. On the flip side, we establish a regret lower bound showing that the exponential dependence on |β| and H is unavoidable for any algorithm with an tildeO(√T) regret (even when the risk objective is on the same scale as the original reward), thus certifying the near-optimality of the proposed algorithms. Our results demonstrate that incorporating risk awareness into reinforcement learning necessitates an exponential cost in |β| and H, which quantifies the fundamental tradeoff between risk sensitivity (related to aleatoric uncertainty) and sample efficiency (related to epistemic uncertainty). To the best of our knowledge, this is the first regret analysis of risk-sensitive reinforcement learning with the exponential utility.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 11; 2023: 9; 2024: 12; 2025: 8; 2026: 9.

57 papers.

Sort:

By area:

  1. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  2. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  3. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
  4. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  5. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  6. 2021Learning Stochastic Shortest Path with Linear Function Approximationcited by 23
  7. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 21
  8. 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 19
  9. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 19
  10. 2021Robust Risk-Aware Reinforcement Learningcited by 17
  11. 2022Learn to Match with No Regret: Reinforcement Learning in Markov Matching Marketscited by 16
  12. 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
38 more
  1. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
  2. 2021Derivative-Free Policy Optimization for Linear Risk-Sensitive and Robust Control Design: Implicit Regularization and Sample Complexitycited by 15
  3. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  4. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 15
  5. 2021Convergence and Alignment of Gradient Descent with Random Backpropagation Weightscited by 14
  6. 2023Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalentscited by 14
  7. 2021Risk-Aware Transfer in Reinforcement Learning using Successor Featurescited by 12
  8. 2024RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learningcited by 8
  9. 2022Complex behavior from intrinsic motivation to occupy action-state path spacecited by 7
  10. 2023Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measurescited by 7
  11. 2023Risk-sensitive Markov Decision Process and Learning under General Utility Functionscited by 7
  12. 2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shiftscited by 6
  13. 2024Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysiscited by 5
  14. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 5
  15. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learningcited by 5
  16. 2022RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Riskcited by 4
  17. 2022Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, and Separation Designcited by 4
  18. 2025Tail-Risk-Safe Monte Carlo Tree Search under PAC-Level Guaranteescited by 4
  19. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximationcited by 3
  20. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 3
  21. 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Modelcited by 3
  22. 2022Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learningcited by 2
  23. 2023Is Risk-Sensitive Reinforcement Learning Properly Resolved?cited by 2
  24. 2025Risk-sensitive Reinforcement Learning Based on Convex Scoring Functionscited by 2
  25. 2025Risk-aware Direct Preference Optimization under Nested Risk Measurecited by 2
  26. 2022Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learningcited by 1
  27. 2022Beyond Bayes-optimality: meta-learning what you know you don't knowcited by 1
  28. 2024TRAM: Test-Time Risk Adaptation with Mixture of Agentscited by 1
  29. 2025Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalentscited by 1
  30. 2026RE-SAC: Disentangling aleatoric and epistemic risks in bus fleet control: A stable and robust ensemble DRL approachcited by 1
  31. 2026Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximationcited by 1
  32. 2023On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processescited by 0
  33. 2024Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observationcited by 0
  34. 2024Taming Equilibrium Bias in Risk-Sensitive Multi-Agent Reinforcement Learningcited by 0
  35. 2024Investigating the Impact of Model Complexity in Large Language Modelscited by 0
  36. 2025Risk-Averse Reinforcement Learning with Itakura-Saito Losscited by 0
  37. 2025Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inferencecited by 0
  38. 2025Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selectioncited by 0

50 most cited of 57 All 57 (JSON)