mlsanityData to 14 Oct 2026

Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning

arXiv 2206.07376 ↗First posted 15 Jun 2022cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Keeping risk under control is often more crucial than maximizing expected rewards in real-world decision-making situations, such as finance, robotics, autonomous driving, etc. The most natural choice of risk measures is variance, which penalizes the upside volatility as much as the downside part. Instead, the (downside) semivariance, which captures the negative deviation of a random variable under its mean, is more suitable for risk-averse proposes. This paper aims at optimizing the mean-semivariance (MSV) criterion in reinforcement learning w.r.t. steady reward distribution. Since semivariance is time-inconsistent and does not satisfy the standard Bellman equation, the traditional dynamic programming methods are inapplicable to MSV problems directly. To tackle this challenge, we resort to Perturbation Analysis (PA) theory and establish the performance difference formula for MSV. We reveal that the MSV problem can be solved by iteratively solving a sequence of RL problems with a policy-dependent reward function. Further, we propose two on-policy algorithms based on the policy gradient theory and the trust region method. Finally, we conduct diverse experiments from simple bandit problems to continuous control tasks in MuJoCo, which demonstrate the effectiveness of our proposed methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 73 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

24 of 73 references.

References · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2016OpenAI Gymcited by 1,736
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  6. 2017Constrained Policy Optimizationcited by 983
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  8. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 337
  9. 2018Maximum a Posteriori Policy Optimisationcited by 328
  10. 2019Challenges of Real-World Reinforcement Learningcited by 291
  11. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  12. 2015Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approachcited by 170
12 more
  1. 2014Optimizing the CVaR via Samplingcited by 99
  2. 2014Algorithms for CVaR Optimization in MDPscited by 99
  3. 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
  4. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 58
  5. 2019Risk-Averse Trust Region Optimization for Reward-Volatility Reductioncited by 54
  6. 2021On-Policy Deep Reinforcement Learning for the Average-Reward Criterioncited by 30
  7. 2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
  8. 2020Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learningcited by 28
  9. 2021Average-Reward Reinforcement Learning with Trust Region Methodscited by 19
  10. 2018A Block Coordinate Ascent Algorithm for Mean-Variance Optimizationcited by 11
  11. 2020Risk-Sensitive Markov Decision Processes with Combined Metrics of Mean and Variancecited by 9
  12. 2022A unified algorithm framework for mean-variance optimization in discounted Markov decision processescited by 3

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Measures of Variability for Risk-averse Policy Gradientcited by 0