mlsanityData to 14 Oct 2026

Boosting CVaR Policy Optimization with Quantile Gradients

arXiv 2601.22100 ↗First posted 29 Jan 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Optimizing Conditional Value-at-risk (CVaR) using policy gradient (a.k.a CVaR-PG) faces significant challenges of sample inefficiency. This inefficiency stems from the fact that it focuses on tail-end performance and overlooks many sampled trajectories. We address this problem by augmenting CVaR with an expected quantile term. Quantile optimization admits a dynamic programming formulation that leverages all sampled data, thus improves sample efficiency. This does not alter the CVaR objective since CVaR corresponds to the expectation of quantile over the tail. Empirical results in domains with verifiable risk-averse behavior show that our algorithm within the Markovian policy class substantially improves upon CVaR-PG and consistently outperforms other existing methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 29 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

15 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Return Capping: Sample-Efficient CVaR Policy Gradient Optimisation29 Apr
  2. 2024Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis31 Oct
  3. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization17 Mar

Other references · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2016OpenAI Gymcited by 1,736
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  6. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  7. 2014Optimizing the CVaR via Samplingcited by 99
  8. 2022Efficient Risk-Averse Reinforcement Learningcited by 43
  9. 2023On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processescited by 18
  10. 2023An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradientcited by 12
  11. 2022Quantile-Based Policy Optimization for Reinforcement Learningcited by 8
  12. 2024Risk-averse Total-reward MDPs with ERM and EVaRcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Long-Run Conditional Value-at-Risk Reinforcement Learning
  2. 2026Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity
  3. 2026Actor-Critic Algorithm for Dynamic Expectile and CVaR
  4. 2025Measures of Variability for Risk-averse Policy Gradient
  5. 2025Improving Value Estimation Critically Enhances Vanilla Policy Gradient
  6. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning
  7. 2026Variance-Reduced Policy Gradient is Nearly Minimax-Optimal
  8. 2025Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Model