mlsanityData to 14 Oct 2026

Risk-sensitive control as inference with R'enyi divergence

arXiv 2411.01827 ↗First posted 4 Nov 2024cs.LG cs.SY eess.SYcited by 3

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper introduces the risk-sensitive control as inference (RCaI) that extends CaI by using R'enyi divergence variational inference. RCaI is shown to be equivalent to log-probability regularized risk-sensitive control, which is an extension of the maximum entropy (MaxEnt) control. We also prove that the risk-sensitive optimal policy can be obtained by solving a soft Bellman equation, which reveals several equivalences between RCaI, MaxEnt control, the optimal posterior for CaI, and linearly-solvable control. Moreover, based on RCaI, we derive the risk-sensitive reinforcement learning (RL) methods: the policy gradient and the soft actor-critic. As the risk-sensitivity parameter vanishes, we recover the risk-neutral CaI and RL, which means that RCaI is a unifying framework. Furthermore, we give another risk-sensitive generalization of the MaxEnt control using R'enyi entropy regularization. We show that in both of our extensions, the optimal policies have the same structure even though the derivations are very different.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 51 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

25 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts15 Feb

Other references · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  5. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  6. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  7. 2017Advances in Variational Inferencecited by 236
  8. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  9. 2018Learning to Walk via Deep Reinforcement Learningcited by 158
  10. 2013Risk-sensitive Reinforcement Learningcited by 130
  11. 2016R'enyi Divergence Variational Inferencecited by 104
  12. 2020Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errorscited by 85
12 more
  1. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  2. 2019Variational Inference MPC for Bayesian Model-based Reinforcement Learningcited by 39
  3. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
  4. 2020Stein Variational Model Predictive Controlcited by 30
  5. 2019Entropic Risk Measure in Policy Searchcited by 19
  6. 2021Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigationcited by 15
  7. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  8. 2021Variational Inference MPC using Tsallis Divergencecited by 11
  9. 2020Variational Model-based Policy Optimizationcited by 7
  10. 2022Kullback-Leibler control for discrete-time nonlinear systems on continuous spacescited by 6
  11. 2022Maximum entropy optimal density control of discrete-time linear systems and Schr"odinger bridgescited by 6
  12. 2023Maximum Entropy Density Control of Discrete-Time Linear Systems with Quadratic Costcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Maximum Likelihood Reinforcement Learningcited by 36
  2. 2025Unifying Entropy Regularization in Optimal Control: From and Back to Classical Objectives via Iterated Soft Policies and Path Integral Solutionscited by 0
  3. 2026The Risk-Sensitive Schr"odinger Bridge: Is Not a KL Projectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents
  2. 2026Risk-Sensitive Exponential Actor Critic
  3. 2026Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning
  4. 2024On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learning
  5. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
  6. 2024Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty
  7. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation
  8. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning