# Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning

arXiv: https://arxiv.org/abs/2004.10888 · HTML: https://mlsanity.com/paper/2004.10888/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2020.

## Built on by (most cited first, 28 of 28)

- [Average-Reward Off-Policy Policy Evaluation with Function Approximation](https://mlsanity.com/paper/2101.02808/) (2021)
- [Challenging Common Assumptions in Convex Reinforcement Learning](https://mlsanity.com/paper/2202.01511/) (2022)
- [A Game-Theoretic Framework for Managing Risk in Multi-Agent Systems](https://mlsanity.com/paper/2205.15434/) (2022)
- [Risk-Aware Transfer in Reinforcement Learning using Successor Features](https://mlsanity.com/paper/2105.14127/) (2021)
- [An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient](https://mlsanity.com/paper/2307.08873/) (2023)
- [Policy Gradient Bayesian Robust Optimization for Imitation Learning](https://mlsanity.com/paper/2106.06499/) (2021)
- [Risk-Sensitive Policy with Distributional Reinforcement Learning](https://mlsanity.com/paper/2212.14743/) (2022)
- [Risk-Sensitive Soft Actor-Critic for Robust Deep Reinforcement Learning under Distribution Shifts](https://mlsanity.com/paper/2402.09992/) (2024)
- [Tractable Equilibrium Computation in Markov Games through Risk Aversion](https://mlsanity.com/paper/2406.14156/) (2024)
- [A unified algorithm framework for mean-variance optimization in discounted Markov decision processes](https://mlsanity.com/paper/2201.05737/) (2022)
- [Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control](https://mlsanity.com/paper/2207.13730/) (2022)
- [Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion](https://mlsanity.com/paper/2310.16546/) (2023)
- [Average Reward Reinforcement Learning for Wireless Radio Resource Management](https://mlsanity.com/paper/2501.06700/) (2025)
- [Risk-sensitive Reinforcement Learning Based on Convex Scoring Functions](https://mlsanity.com/paper/2505.04553/) (2025)
- [Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning](https://mlsanity.com/paper/2507.03900/) (2025)
- [LAD: Learning Advantage Distribution for Reasoning](https://mlsanity.com/paper/2602.20132/) (2026)
- [SAT: Sequential Agent Tuning for Coordinator Free Plug and Play Multi-LLM Training with Monotonic Improvement Guarantees](https://mlsanity.com/paper/2605.05216/) (2026)
- [Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2206.07376/) (2022)
- [Global Algorithms for Mean-Variance Optimization in Markov Decision Processes](https://mlsanity.com/paper/2302.13710/) (2023)
- [Bootstrapping Expectiles in Reinforcement Learning](https://mlsanity.com/paper/2406.04081/) (2024)
- [A Finite-Sample Analysis of an Actor-Critic Algorithm for Mean-Variance Optimization in a Discounted MDP](https://mlsanity.com/paper/2406.07892/) (2024)
- [TRAM: Test-Time Risk Adaptation with Mixture of Agents](https://mlsanity.com/paper/2408.08812/) (2024)
- [STOPS: Short-Term-based Volatility-controlled Policy Search and its Global Convergence](https://mlsanity.com/paper/2201.09857/) (2022)
- [Measures of Variability for Risk-averse Policy Gradient](https://mlsanity.com/paper/2504.11412/) (2025)
- [Risk-Averse Total-Reward Reinforcement Learning](https://mlsanity.com/paper/2506.21683/) (2025)
- [Sharpe Ratio Optimization in Markov Decision Processes](https://mlsanity.com/paper/2509.00793/) (2025)
- [Risk-Sensitive Exponential Actor Critic](https://mlsanity.com/paper/2602.07202/) (2026)
- [A Noise-Robust Elicit-to-Optimize Framework for Distortion Riskmetrics via Inverse Reinforcement Learning](https://mlsanity.com/paper/2607.14373/) (2026)

## Abstract (arXiv)

We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in that any policy evaluation method and risk-neutral control method can be dropped in for risk-averse control off the shelf, in both on- and off-policy settings. This flexibility reduces the gap between risk-neutral control and risk-averse control and is achieved by working on a novel augmented MDP directly. We propose risk-averse TD3 as an example instantiating MVPI, which outperforms vanilla TD3 and many previous risk-averse control methods in challenging Mujoco robot simulation tasks under a risk-aware performance metric. This risk-averse TD3 is the first to introduce deterministic policies and off-policy learning into risk-averse reinforcement learning, both of which are key to the performance boost we show in Mujoco domains.