mlsanityData to 14 Oct 2026

Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts

arXiv 2402.09992 ↗First posted 15 Feb 2024cs.LG cs.SY eess.SYcited by 6

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We study the robustness of deep reinforcement learning algorithms against distribution shifts within contextual multi-stage stochastic combinatorial optimization problems from the operations research domain. In this context, risk-sensitive algorithms promise to learn robust policies. While this field is of general interest to the reinforcement learning community, most studies up-to-date focus on theoretical results rather than real-world performance. With this work, we aim to bridge this gap by formally deriving a novel risk-sensitive deep reinforcement learning algorithm while providing numerical evidence for its efficacy. Specifically, we introduce discrete Soft Actor-Critic for the entropic risk measure by deriving a version of the Bellman equation for the respective Q-values. We establish a corresponding policy improvement result and infer a practical algorithm. We introduce an environment that represents typical contextual multi-stage stochastic combinatorial optimization problems and perform numerical experiments to empirically validate our algorithm's robustness against realistic distribution shifts, without compromising performance on the training distribution. We show that our algorithm is superior to risk-neutral Soft Actor-Critic as well as to two benchmark approaches for robust deep reinforcement learning. Thereby, we provide the first structured analysis on the robustness of reinforcement learning under distribution shifts in the realm of contextual multi-stage stochastic combinatorial optimization problems.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 49 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

24 of 49 references.

References · 24

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  5. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 261
  6. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  7. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  8. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  9. 2018Learning to Walk via Deep Reinforcement Learningcited by 158
  10. 2014Algorithms for CVaR Optimization in MDPscited by 99
  11. 2015Policy Gradient for Coherent Risk Measurescited by 76
  12. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 72
12 more
  1. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  2. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
  3. 2021A Deep Value-network Based Approach for Multi-Driver Order Dispatchingcited by 45
  4. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  5. 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
  6. 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
  7. 2020Improving Robustness via Risk Averse Distributional Reinforcement Learningcited by 28
  8. 2021Graph Neural Network Reinforcement Learning for Autonomous Mobility-on-Demand Systemscited by 28
  9. 2014Policy Gradients for CVaR-Constrained MDPscited by 21
  10. 2019Entropic Risk Measure in Policy Searchcited by 19
  11. 2022Hybrid Multi-agent Deep Reinforcement Learning for Autonomous Mobility on Demand Systemscited by 18
  12. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penaltycited by 9
  2. 2024Risk-sensitive control as inference with R'enyi divergencecited by 3
  3. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  4. 2026Reinforcement Learning for Exponential Utility: Algorithms and Convergence in Discounted MDPscited by 1
  5. 2026Robust General Utility for Reinforcement Learningcited by 1
  6. 2025Risk-Averse Reinforcement Learning with Itakura-Saito Losscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safe Langevin Soft Actor Critic
  2. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
  3. 2026Risk-Sensitive Exponential Actor Critic
  4. 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty
  5. 2025Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization
  6. 2026Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
  7. 2024Rethinking Soft Actor-Critic in High-Dimensional Action Spaces: The Cost of Ignoring Distribution Shift
  8. 2024Robust Reinforcement Learning with Dynamic Distortion Risk Measures