mlsanityData to 14 Oct 2026

Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning

arXiv 2507.03900 ↗First posted 5 Jul 2025cs.LG stat.MLcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The development of Distributional Reinforcement Learning (DRL) has introduced a natural way to incorporate risk sensitivity into value-based and actor-critic methods by employing risk measures other than expectation in the value function. While this approach is widely adopted in many online and offline RL algorithms due to its simplicity, the naive integration of risk measures often results in suboptimal policies. This limitation can be particularly harmful in scenarios where the need for effective risk-sensitive policies is critical and worst-case outcomes carry severe consequences. To address this challenge, we propose a novel framework for optimizing static Spectral Risk Measures (SRM), a flexible family of risk measures that generalizes objectives such as CVaR and Mean-CVaR, and enables the tailoring of risk preferences. Our method is applicable to both online and offline RL algorithms. We establish theoretical guarantees by proving convergence in the finite state-action setting. Moreover, through extensive empirical evaluations, we demonstrate that our algorithms consistently outperform existing risk-sensitive methods in both online and offline environments across diverse domains.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 69 references that are in the map. First posted on arXiv 5 Jul 2025.

Builds on

33 of 69 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning30 Nov
  2. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 24

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  9. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  11. 2015Coordinate Descent Algorithmscited by 219
  12. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
12 more
  1. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  2. 2014Optimizing the CVaR via Samplingcited by 99
  3. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  4. 2022Efficient Risk-Averse Reinforcement Learningcited by 43
  5. 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
  6. 2021Reinforcement Learning with Dynamic Convex Risk Measurescited by 21
  7. 2019An intelligent financial portfolio trading strategy using deep Q-learningcited by 14
  8. 2020Minimizing Spectral Risk Measures Applied to Markov Decision Processescited by 12
  9. 2025Optimizing Return Distributions with Distributional Dynamic Programmingcited by 10
  10. 2015Stochastic Optimal Control With Dynamic, Time-Consistent Risk Constraintscited by 6
  11. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learningcited by 6
  12. 2024Spectral-Risk Safe Reinforcement Learning with Convergence Guaranteescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Decoupling Time and Risk: Risk-Sensitive Reinforcement Learning with General Discountingcited by 1
  2. 2026Position: Deployed Reinforcement Learning should be Continualcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation
  2. 2025RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization
  3. 2024On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learning
  4. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
  5. 2026Risk-Sensitive Exponential Actor Critic
  6. 2026Safe RLHF Beyond Expectation: Stochastic Dominance for Universal Spectral Risk Control
  7. 2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts
  8. 2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback