mlsanityData to 14 Oct 2026

Long-Run Conditional Value-at-Risk Reinforcement Learning

arXiv 2603.09734 ↗First posted 10 Mar 2026math.OCcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Conditional value-at-risk (CVaR) is a prominent risk measure in financial engineering, energy systems, and supply chain management. In these domains, Markov decision processes (MDPs) with a long-run CVaR criterion effectively mitigate cost variability over a specified horizon. However, implementing MDPs relies on known transition models, which are typically unavailable in practice. This necessitates a model-free approach to risk-sensitive dynamic optimization. To tackle this challenge, we propose a reinforcement learning algorithm that simultaneously conducts policy evaluation and improvement based on a CVaR-specific Bellman local optimality equation. This algorithm employs a nonparametric incremental learning approach for policy improvement, relying on a single sample trajectory to identify the optimal policy. Under appropriate technical conditions, we prove almost sure convergence of the algorithm and derive its convergence rate. Our analysis reveals that the optimal convergence rate, measured by the mean absolute error of policy estimators, is of order O(1/n). Our main algorithm and results are further extended to solving the mean-CVaR optimization problem. Numerical experiments corroborate these results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 53 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

13 of 53 references.

References · 13

  1. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  2. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  3. 2014Optimizing the CVaR via Samplingcited by 99
  4. 2021Simple Agent, Complex Environment: Efficient Reinforcement Learning with Agent Statescited by 43
  5. 2022Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximationcited by 31
  6. 2014Policy Gradients for CVaR-Constrained MDPscited by 21
  7. 2022A Policy Gradient Algorithm for the Risk-Sensitive Exponential Cost MDPcited by 13
  8. 2020Risk-Sensitive Markov Decision Processes with Combined Metrics of Mean and Variancecited by 9
  9. 2015Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measurescited by 6
  10. 2022Risk-Sensitive Markov Decision Processes with Long-Run CVaR Criterioncited by 4
  11. 2023Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithmscited by 4
  12. 2023Quantile Optimization via Multiple Timescale Local Search for Black-box Functionscited by 4
1 more
  1. 2019A Multi-Level Simulation Optimization Approach for Quantile Functionscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity
  2. 2026Boosting CVaR Policy Optimization with Quantile Gradients
  3. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization
  4. 2024Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk
  5. 2026Actor-Critic Algorithm for Dynamic Expectile and CVaR
  6. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning
  7. 2025Adaptive Insurance Reserving with CVaR-Constrained Reinforcement Learning under Macroeconomic Regimes
  8. 2024Catastrophic-risk-aware reinforcement learning with extreme-value-theory-based policy gradients