mlsanityData to 14 Oct 2026

Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning

arXiv 2407.07631 ↗First posted 10 Jul 2024cs.LG math.OC math.STcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study risk-sensitive reinforcement learning (RL), a crucial field due to its ability to enhance decision-making in scenarios where it is essential to manage uncertainty and minimize potential adverse outcomes. Particularly, our work focuses on applying the entropic risk measure to RL problems. While existing literature primarily investigates the online setting, there remains a large gap in understanding how to efficiently derive a near-optimal policy based on this risk measure using only a pre-collected dataset. We center on the linear Markov Decision Process (MDP) setting, a well-regarded theoretical framework that has yet to be examined from a risk-sensitive standpoint. In response, we introduce two provably sample-efficient algorithms. We begin by presenting a risk-sensitive pessimistic value iteration algorithm, offering a tight analysis by leveraging the structure of the risk-sensitive performance measure. To further improve the obtained bounds, we propose another pessimistic algorithm that utilizes variance information and reference-advantage decomposition, effectively improving both the dependence on the space dimension d and the risk-sensitivity factor. To the best of our knowledge, we obtain the first provably efficient risk-sensitive offline RL algorithms.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 61 of its 91 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

61 of 91 references.

Closest ancestors · 21

Same direction, newest first

  1. 2023Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning2 Oct
  2. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning30 Nov
  3. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation23 Nov
  4. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game31 May
  5. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  6. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  7. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  8. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
  9. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  10. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  11. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  12. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
9 more
  1. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  2. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  5. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  6. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  7. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  8. 2020On Reward-Free Reinforcement Learning with Linear Function Approximation19 Jun
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 40

  1. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  2. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  3. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  4. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  5. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 200
  6. 2013Risk-sensitive Reinforcement Learningcited by 130
  7. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  8. 2021Recent Advances in Reinforcement Learning in Financecited by 110
  9. 2020Provably Efficient Reinforcement Learning for Discounted MDPs with Feature Mappingcited by 102
  10. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 92
  11. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  12. 2020Logarithmic Regret for Reinforcement Learning with Linear Function Approximationcited by 79
28 more
  1. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
  2. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  3. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 52
  4. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  5. 2021First-Order Regret in Reinforcement Learning with Linear Function Approximation: A Robust Estimation Approachcited by 38
  6. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  7. 2020Learning Bounds for Risk-sensitive Learningcited by 33
  8. 2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
  9. 2022Nearly Minimax Optimal Reinforcement Learning with Linear Function Approximationcited by 30
  10. 2021Reward-Free Model-Based Reinforcement Learning with Linear Function Approximationcited by 29
  11. 2023A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processescited by 29
  12. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  13. 2022Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 26
  14. 2020Nonstationary Reinforcement Learning with Linear Function Approximationcited by 23
  15. 2019Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimizationcited by 22
  16. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 22
  17. 2014Policy Gradients for CVaR-Constrained MDPscited by 21
  18. 2021On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Gamecited by 20
  19. 2019Entropic Risk Measure in Policy Searchcited by 19
  20. 2022Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Boundscited by 16
  21. 2023Efficient Model-Free Exploration in Low-Rank MDPscited by 16
  22. 2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedbackcited by 14
  23. 2022A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDPcited by 13
  24. 2023Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RLcited by 13
  25. 2023Risk-sensitive Markov Decision Process and Learning under General Utility Functionscited by 7
  26. 2023Provably Efficient CVaR RL in Low-rank MDPscited by 6
  27. 2020Deep Deterministic Portfolio Optimizationcited by 4
  28. 2023Is Risk-Sensitive Reinforcement Learning Properly Resolved?cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 6
  2. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filteringcited by 2
  3. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learningcited by 0
  4. 2026Pessimistic Risk-Aware Policy Learning in Contextual Banditscited by 0
  5. 2026Long-Term Sequential Decision Making under Riskcited by 0
  6. 2026Fast Rates for Offline Q-Learning with KL Regularizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents
  2. 2025Robust Bayesian Dynamic Programming for On-policy Risk-sensitive Reinforcement Learning
  3. 2024Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis
  4. 2024Provably Efficient Partially Observable Risk-Sensitive Reinforcement Learning with Hindsight Observation
  5. 2024Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk
  6. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
  7. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation
  8. 2025Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents