mlsanityData to 14 Oct 2026

A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimization

arXiv 2403.11062 ↗First posted 17 Mar 2024cs.LG math.OCcited by 7

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement learning algorithms utilizing policy gradients (PG) to optimize Conditional Value at Risk (CVaR) face significant challenges with sample inefficiency, hindering their practical applications. This inefficiency stems from two main facts: a focus on tail-end performance that overlooks many sampled trajectories, and the potential of gradient vanishing when the lower tail of the return distribution is overly flat. To address these challenges, we propose a simple mixture policy parameterization. This method integrates a risk-neutral policy with an adjustable policy to form a risk-averse policy. By employing this strategy, all collected trajectories can be utilized for policy updating, and the issue of vanishing gradients is counteracted by stimulating higher returns through the risk-neutral component, thus lifting the tail and preventing flatness. Our empirical study reveals that this mixture parameterization is uniquely effective across a variety of benchmark domains. Specifically, it excels in identifying risk-averse CVaR policies in some Mujoco environments where the traditional CVaR-PG fails to learn a reasonable policy.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 58 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

30 of 58 references.

References · 30

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  8. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  9. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  10. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  11. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  12. 2021Offline RL Without Off-Policy Evaluationcited by 151
18 more
  1. 2014Optimizing the CVaR via Samplingcited by 99
  2. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
  3. 2019Worst Cases Policy Gradientscited by 52
  4. 2020Inverse Constrained Reinforcement Learningcited by 46
  5. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  6. 2022Efficient Risk-Averse Reinforcement Learningcited by 43
  7. 2022Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Riskcited by 41
  8. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  9. 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
  10. 2022Provably Efficient Risk-Sensitive Reinforcement Learning: Iterated CVaR and Worst Pathcited by 22
  11. 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 20
  12. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  13. 2023On Dynamic Programming Decompositions of Static Risk Measures in Markov Decision Processescited by 18
  14. 2020Continuous Action Reinforcement Learning from a Mixture of Interpretable Expertscited by 17
  15. 2020Implicit Distributional Reinforcement Learningcited by 16
  16. 2021Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learningcited by 15
  17. 2023An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradientcited by 12
  18. 2022Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probabilitycited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  2. 2025Return Capping: Sample-Efficient CVaR Policy Gradient Optimisationcited by 6
  3. 2026Actor-Critic Algorithm for Dynamic Expectile and CVaRcited by 2
  4. 2025Measures of Variability for Risk-averse Policy Gradientcited by 0
  5. 2025RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularizationcited by 0
  6. 2025Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inferencecited by 0
  7. 2026Boosting CVaR Policy Optimization with Quantile Gradientscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Long-Run Conditional Value-at-Risk Reinforcement Learning
  2. 2024On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learning
  3. 2025Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Model
  4. 2026Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity
  5. 2024Towards Efficient Risk-Sensitive Policy Gradient: An Iteration Complexity Analysis
  6. 2026Revisiting Mixture Policies in Entropy-Regularized Actor-Critic
  7. 2024Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate
  8. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning