mlsanityData to 14 Oct 2026

Efficient Risk-Averse Reinforcement Learning

arXiv 2205.05138 ↗First posted 10 May 2022cs.LGcited by 43

Not on the map. Why?

Cited by 22 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In risk-averse reinforcement learning (RL), the goal is to optimize some risk measure of the returns. A risk measure often focuses on the worst returns out of the agent's experience. As a result, standard methods for risk-averse RL often ignore high-return strategies. We prove that under certain conditions this inevitably leads to a local-optimum barrier, and propose a soft risk mechanism to bypass it. We also devise a novel Cross Entropy module for risk sampling, which (1) preserves risk aversion despite the soft risk; (2) independently improves sample efficiency. By separating the risk aversion of the sampler and the optimizer, we can sample episodes with poor conditions, yet optimize with respect to successful strategies. We combine these two concepts in CeSoR - Cross-entropy Soft-Risk optimization algorithm - which can be applied on top of any risk-averse policy gradient (PG) method. We demonstrate improved risk aversion in maze navigation, autonomous driving, and resource allocation benchmarks, including in scenarios where standard risk-averse PG completely fails.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 26 of its 54 references that are in the map. First posted on arXiv 10 May 2022.

Builds on

26 of 54 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  4. 2017Robust Adversarial Reinforcement Learningcited by 451
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  6. 2015Contextual Markov Decision Processescited by 220
  7. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
  8. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
  9. 2014Optimizing the CVaR via Samplingcited by 99
  10. 2014Algorithms for CVaR Optimization in MDPscited by 99
  11. 2015Policy Gradient for Coherent Risk Measurescited by 76
  12. 2019Being Optimistic to Be Conservative: Quickly Learning a CVaR Policycited by 63
14 more
  1. 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 59
  2. 2019Worst Cases Policy Gradientscited by 52
  3. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 43
  4. 2019Adaptive Sampling for Stochastic Risk-Averse Learningcited by 39
  5. 2021RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agentscited by 31
  6. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 29
  7. 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
  8. 2021On the Convergence and Optimality of Policy Gradient for Markov Coherent Riskcited by 20
  9. 2019Learning Robust Options by Conditional Value at Risk Optimizationcited by 16
  10. 2020Option Hedging with Risk Averse Reinforcement Learningcited by 13
  11. 2017Risk-sensitive Inverse Reinforcement Learning via Semi- and Non-Parametric Methodscited by 12
  12. 2018A Block Coordinate Ascent Algorithm for Mean-Variance Optimizationcited by 11
  13. 2021Reinforcement Learning for Datacenter Congestion Controlcited by 11
  14. 2021CEM-GD: Cross-Entropy Method with Gradient Descent Planner for Model-Based Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 10; 2025: 16; 2026: 8.

43 papers.

Sort:

By area:

  1. 2024A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalentscited by 16
  2. 2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
  3. 2023Train Hard, Fight Easy: Robust Meta Reinforcement Learningcited by 12
  4. 2023An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradientcited by 12
  5. 2024CIMRL: Combining IMitation and Reinforcement Learning for Safe Autonomous Drivingcited by 12
  6. 2025Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environmentscited by 12
  7. 2023ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantagescited by 10
  8. 2023Distributional Model Equivalence for Risk-Sensitive Reinforcement Learningcited by 9
  9. 2023Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernelcited by 8
  10. 2024A Simple Mixture Policy Parameterization for Improving Sample Efficiency of CVaR Optimizationcited by 7
  11. 2025Risk-Averse Finetuning of Large Language Modelscited by 7
  12. 2025Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learningcited by 6
31 more
  1. 2025Return Capping: Sample-Efficient CVaR Policy Gradient Optimisationcited by 6
  2. 2023Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theorycited by 5
  3. 2025Safety Representations for Safer Policy Learningcited by 4
  4. 2025Optimistic Exploration for Risk-Averse Constrained Reinforcement Learningcited by 3
  5. 2025Bipedalism for Quadrupedal Robots: Versatile Loco-Manipulation through Risk-Adaptive Reinforcement Learningcited by 3
  6. 2024UOEP: User-Oriented Exploration Policy for Enhancing Long-Term User Experiences in Recommender Systemscited by 2
  7. 2024Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Riskcited by 2
  8. 2024Catastrophic-risk-aware reinforcement learning with extreme-value-theory-based policy gradientscited by 2
  9. 2025Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learningcited by 2
  10. 2026Actor-Critic Algorithm for Dynamic Expectile and CVaRcited by 2
  11. 2023Distributional Reinforcement Learning with Dual Expectile-Quantile Regressioncited by 1
  12. 2024Bootstrapping Expectiles in Reinforcement Learningcited by 1
  13. 2025Risk-Sensitive Agent Compositionscited by 1
  14. 2023Individualized Dosing Dynamics via Neural Eigen Decompositioncited by 0
  15. 2024Bellman Unbiasedness: Toward Provably Efficient Distributional Reinforcement Learning with General Value Function Approximationcited by 0
  16. 2024Risk-averse learning with delayed feedbackcited by 0
  17. 2024Neuromodulated Meta-Learningcited by 0
  18. 2025Risk-averse policies for natural gas futures trading using distributional reinforcement learningcited by 0
  19. 2025Measures of Variability for Risk-averse Policy Gradientcited by 0
  20. 2025Online Episodic Convex Reinforcement Learningcited by 0
  21. 2025Risk-Averse Total-Reward Reinforcement Learningcited by 0
  22. 2025FR-LUX: Friction-Aware, Regime-Conditioned Policy Optimization for Implementable Portfolio Managementcited by 0
  23. 2025Risk-Aware Reinforcement Learning with Bandit-Based Adaptation for Quadrupedal Locomotioncited by 0
  24. 2025Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inferencecited by 0
  25. 2026Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertaintycited by 0
  26. 2026Risk-Aware Human-in-the-Loop Framework with Adaptive Intrusion Response for Autonomous Vehiclescited by 0
  27. 2026Boosting CVaR Policy Optimization with Quantile Gradientscited by 0
  28. 2026Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinitycited by 0
  29. 2026The Price of Paranoia: Robust Risk-Sensitive Cooperation in Non-Stationary Multi-Agent Reinforcement Learningcited by 0
  30. 2026Predictive Safety Curricula for Robust Legged Locomotioncited by 0
  31. 2026Bridging Risk Approximation Gaps in Model Predictive Task Sampling via In-Context Modelingcited by 0