mlsanityData to 14 Oct 2026

Continuous-time Risk-sensitive Reinforcement Learning via Quadratic Variation Penalty

arXiv 2404.12598 ↗First posted 19 Apr 2024cs.LG cs.SY eess.SYcited by 9

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

This paper studies continuous-time risk-sensitive reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation with the exponential-form objective. The risk-sensitive objective arises either as the agent's risk attitude or as a distributionally robust approach against the model uncertainty. Owing to the martingale perspective in Jia and Zhou (J Mach Learn Res 24(161): 1--61, 2023) the risk-sensitive RL problem is shown to be equivalent to ensuring the martingale property of a process involving both the value function and the q-function, augmented by an additional penalty term: the quadratic variation of the value process, capturing the variability of the value-to-go along the trajectory. This characterization allows for the straightforward adaptation of existing RL algorithms developed for non-risk-sensitive scenarios to incorporate risk sensitivity by adding the realized variance of the value process. Additionally, I highlight that the conventional policy gradient representation is inadequate for risk-sensitive problems due to the nonlinear nature of quadratic variation; however, q-learning offers a solution and extends to infinite horizon settings. Finally, I prove the convergence of the proposed algorithm for Merton's investment problem and quantify the impact of temperature parameter on the behavior of the learning procedure. I also conduct simulation experiments to demonstrate how risk-sensitive RL improves the finite-sample performance in the linear-quadratic control problem.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 64 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

19 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts15 Feb

Other references · 18

  1. 2018Is Q-learning Provably Efficient?cited by 487
  2. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  3. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  4. 2021Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithmscited by 101
  5. 2022q-Learning in Continuous Timecited by 91
  6. 2021Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approachcited by 84
  7. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  8. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
  9. 2020Entropy Regularization for Mean Field Games with Learningcited by 58
  10. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  11. 2023A Finite Sample Complexity Bound for Distributionally Robust Q-learningcited by 36
  12. 2021Exploratory HJB equations and their convergencecited by 32
6 more
  1. 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 31
  2. 2018Distributionally Robust Mean-Variance Portfolio Selection with Wasserstein Distancescited by 25
  3. 2023Actor-Critic learning for mean-field control in continuous timecited by 22
  4. 2023Continuous-time q-learning for mean-field control problemscited by 22
  5. 2023Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalentscited by 15
  6. 2023Risk-sensitive Markov Decision Process and Learning under General Utility Functionscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Accuracy of Discretely Sampled Stochastic Policies in Continuous-time Reinforcement Learningcited by 13
  2. 2026Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundationscited by 6
  3. 2026Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithmscited by 4
  4. 2025Risk-sensitive Reinforcement Learning Based on Convex Scoring Functionscited by 2
  5. 2025A Reinforcement Learning Framework for Some Singular Stochastic Control Problemscited by 2
  6. 2025Reinforcement learning for irreversible reinsurance problems: the randomized singular control approachcited by 1
  7. 2026Exploratory Randomization for Discrete-Time Risk-Sensitive Benchmarked Investment Management with Reinforcement Learningcited by 1
  8. 2026Reinforcement Learning for Risk-Sensitive Investment Management: a Free Energy--Entropy Duality Approachcited by 0
  9. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Risk-Sensitive Q-Learning in Continuous Time with Application to Dynamic Portfolio Selection
  2. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
  3. 2024Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation
  4. 2024Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis
  5. 2024On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learning
  6. 2025Robust Bayesian Dynamic Programming for On-policy Risk-sensitive Reinforcement Learning
  7. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents
  8. 2023Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback