mlsanityData to 14 Oct 2026

Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning

arXiv 2111.03947 ↗First posted 6 Nov 2021cs.LG math.OC stat.MLcited by 43

Not on the map. Why?

Abstract (arXiv)

We study risk-sensitive reinforcement learning (RL) based on the entropic risk measure. Although existing works have established non-asymptotic regret guarantees for this problem, they leave open an exponential gap between the upper and lower bounds. We identify the deficiencies in existing algorithms and their analysis that result in such a gap. To remedy these deficiencies, we investigate a simple transformation of the risk-sensitive Bellman equations, which we call the exponential Bellman equation. The exponential Bellman equation inspires us to develop a novel analysis of Bellman backup procedures in risk-sensitive RL algorithms, and further motivates the design of a novel exploration mechanism. We show that these analytic and algorithmic innovations together lead to improved regret upper bounds over existing ones.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 20 of its 44 references that are in the map. First posted on arXiv 6 Nov 2021.

Builds on

20 of 44 references.

References · 20

  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  4. 2018Is Q-learning Provably Efficient?cited by 487
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  6. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  7. 2013Risk-sensitive Reinforcement Learningcited by 130
  8. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 88
  9. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  10. 2018An Analysis of Categorical Distributional Reinforcement Learningcited by 68
  11. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
  12. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
8 more
  1. 2020Multiple Descent: Design Your Own Generalization Curvecited by 45
  2. 2019Minimax Regret of Switching-Constrained Online Convex Optimization: No Phase Transitioncited by 17
  3. 2018Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agentscited by 15
  4. 2021Convergence and Alignment of Gradient Descent with Random Backpropagation Weightscited by 14
  5. 2018On the Landscape of Synchronization Networks: A Perspective from Nonconvex Optimizationcited by 11
  6. 2018Stochastic Approximation for Risk-aware Markov Decision Processescited by 10
  7. 2020Meta Learning in the Continuous Time Limitcited by 10
  8. 2021Infinite-Horizon Offline Reinforcement Learning with Linear Function Approximation: Curse of Dimensionality and Algorithmcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 11; 2023: 7; 2024: 10; 2025: 7; 2026: 8.

43 papers.

Sort:

By area:

  1. 2022Robust Reinforcement Learning using Offline Datacited by 96
  2. 2022Efficient Risk-Averse Reinforcement Learningcited by 43
  3. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  4. 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
  5. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 22
  6. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 17
  7. 2022Learn to Match with No Regret: Reinforcement Learning in Markov Matching Marketscited by 16
  8. 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
  9. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 16
  10. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  11. 2023Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalentscited by 15
  12. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 9
31 more
  1. 2023Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measurescited by 8
  2. 2023Risk-sensitive Markov Decision Process and Learning under General Utility Functionscited by 7
  3. 2024Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Modelcited by 7
  4. 2022A Risk-Sensitive Approach to Policy Optimizationcited by 6
  5. 2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shiftscited by 6
  6. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learningcited by 6
  7. 2025Recursive Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Modelcited by 5
  8. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  9. 2022Non-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, and Separation Designcited by 4
  10. 2024Risk-averse Total-reward MDPs with ERM and EVaRcited by 4
  11. 2023Policy Gradient with Tree Expansioncited by 3
  12. 2024Risk-sensitive control as inference with R'enyi divergencecited by 3
  13. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  14. 2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
  15. 2022Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learningcited by 2
  16. 2025Risk-sensitive Reinforcement Learning Based on Convex Scoring Functionscited by 2
  17. 2022A Survey of Learning on Small Data: Generalization, Optimization, and Challengecited by 1
  18. 2023Learning Rewards to Optimize Global Performance Metrics in Deep Reinforcement Learningcited by 1
  19. 2024Risk-Sensitive Multi-Agent Reinforcement Learning in Network Aggregative Markov Gamescited by 1
  20. 2025Optimism as Risk-Seeking in Multi-Agent Reinforcement Learningcited by 1
  21. 2026Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPscited by 1
  22. 2026On the Sample Complexity of Discounted Reinforcement Learning with Optimized Certainty Equivalentscited by 1
  23. 2024Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observationcited by 0
  24. 2024Taming Equilibrium Bias in Risk-Sensitive Multi-Agent Reinforcement Learningcited by 0
  25. 2025Risk-averse policies for natural gas futures trading using distributional reinforcement learningcited by 0
  26. 2025Risk-Averse Reinforcement Learning with Itakura-Saito Losscited by 0
  27. 2025Risk-Averse Total-Reward Reinforcement Learningcited by 0
  28. 2026Progressive Power Homotopy for Non-convex Optimizationcited by 0
  29. 2026Risk-Sensitive Exponential Actor Criticcited by 0
  30. 2026Portfolio Reinforcement Learning with Scenario-Context Rolloutcited by 0
  31. 2026Near-Optimal Sample Complexity for Recursive Entropic Risk Reinforcement Learning with a Generative Modelcited by 0