mlsanityData to 14 Oct 2026

Actor-Critic Algorithm for Dynamic Expectile and CVaR

arXiv 2605.07857 ↗First posted 8 May 2026cs.LGcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Optimizing dynamic risk with stochastic policies is challenging in both policy updates and value learning. The former typically requires transition perturbation, while the latter may rely on model-based approaches. To address these challenges, we propose a surrogate policy gradient without transition perturbation under softmax policy parameterization. We further develop model-free value learning methods for dynamic expectile and conditional value-at-risk by leveraging elicitability. Finally, inspired by Expected SARSA and Expected Policy Gradient, a model-free off-policy actor-critic algorithm is constructed. Empirical results in domains with verifiable risk-averse behavior show that our algorithm can learn risk-averse policy and consistently outperforms other existing methods.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 37 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

18 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation29 Apr
  2. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization17 Mar

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2016OpenAI Gymcited by 1,736
  4. 2014Convex Optimization: Algorithms and Complexitycited by 575
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  6. 2013Risk-sensitive Reinforcement Learningcited by 130
  7. 2015Policy Gradient for Coherent Risk Measurescited by 76
  8. 2022Efficient Risk-Averse Reinforcement Learningcited by 43
  9. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  10. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 23
  11. 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 20
  12. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 17
4 more
  1. 2022Conditionally Elicitable Dynamic Risk Measures for Deep Reinforcement Learningcited by 9
  2. 2023On the Global Convergence of Risk-Averse Natural Policy Gradient Methods with Expected Conditional Risk Measurescited by 7
  3. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
  4. 2026Dynamic Programming for Epistemic Uncertainty in Markov Decision Processescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Online Policy Evaluation for MDPs with Dynamic UBSR Measurescited by 0
  2. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Long-Run Conditional Value-at-Risk Reinforcement Learning
  2. 2026CARE-VI: Conservative Adaptive Reliability Estimation for Value Improvement in Off-Policy Actor-Critic Learning
  3. 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty
  4. 2026Boosting CVaR Policy Optimization with Quantile Gradients
  5. 2026Risk-Sensitive Exponential Actor Critic
  6. 2026Safe Langevin Soft Actor Critic
  7. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning
  8. 2025Moderate Actor-Critic Methods: Controlling Overestimation Bias via Expectile Loss