# Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory

arXiv: https://arxiv.org/abs/2308.13011 · HTML: https://mlsanity.com/paper/2308.13011/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Risk-Constrained Reinforcement Learning with Percentile Risk Criteria](https://mlsanity.com/paper/1512.01629/) (2015)
- [Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach](https://mlsanity.com/paper/1506.02188/) (2015)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Extreme Q-Learning: MaxEnt RL without Entropy](https://mlsanity.com/paper/2301.02328/) (2023)
- [Optimizing the CVaR via Sampling](https://mlsanity.com/paper/1404.3862/) (2014)
- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning](https://mlsanity.com/paper/1711.06782/) (2017)
- [Conservative Offline Distributional Reinforcement Learning](https://mlsanity.com/paper/2107.06106/) (2021)
- [Risk-Averse Offline Reinforcement Learning](https://mlsanity.com/paper/2102.05371/) (2021)
- [Risk Averse Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1904.00511/) (2019)
- [Worst Cases Policy Gradients](https://mlsanity.com/paper/1911.03618/) (2019)
- [Efficient Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2205.05138/) (2022)
- [Distributionally Robust Reinforcement Learning](https://mlsanity.com/paper/1902.08708/) (2019)
- [Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk](https://mlsanity.com/paper/2206.04436/) (2022)
- [Learning Robust Options by Conditional Value at Risk Optimization](https://mlsanity.com/paper/1905.09191/) (2019)
- [An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient](https://mlsanity.com/paper/2307.08873/) (2023)
- [Understanding the Under-Coverage Bias in Uncertainty Estimation](https://mlsanity.com/paper/2106.05515/) (2021)
- [Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning](https://mlsanity.com/paper/2301.05664/) (2023)
- [RASR: Risk-Averse Soft-Robust MDPs with EVaR and Entropic Risk](https://mlsanity.com/paper/2209.04067/) (2022)
- [Deep Reinforcement Learning for Equal Risk Pricing and Hedging under Dynamic Expectile Risk Measures](https://mlsanity.com/paper/2109.04001/) (2021)
- [Estimation and uncertainty quantification for extreme quantile regions](https://mlsanity.com/paper/1904.08251/) (2019)
- [Risk-Averse Action Selection Using Extreme Value Theory Estimates of the CVaR](https://mlsanity.com/paper/1912.01718/) (2019)

## Built on by (most cited first, 5 of 5)

- [Extreme Value Policy Optimization for Safe Reinforcement Learning](https://mlsanity.com/paper/2601.12008/) (2026)
- [A Kolmogorov-Arnold Neural Model for Cascading Extremes](https://mlsanity.com/paper/2505.13370/) (2025)
- [SteinGate: Tail-Sensitive Safe Reinforcement Learning via Stein Discrepancy](https://mlsanity.com/paper/2607.13175/) (2026)
- [Rule-based and reinforcement-learning policies for resource allocation in a pediatric echo lab: a simulation study](https://mlsanity.com/paper/2506.06297/) (2025)
- [Training ML Models with Predictable Failures](https://mlsanity.com/paper/2605.15134/) (2026)

## Abstract (arXiv)

Risk-sensitive reinforcement learning (RL) has garnered significant attention in recent years due to the growing interest in deploying RL agents in real-world scenarios. A critical aspect of risk awareness involves modeling highly rare risk events (rewards) that could potentially lead to catastrophic outcomes. These infrequent occurrences present a formidable challenge for data-driven methods aiming to capture such risky events accurately. While risk-aware RL techniques do exist, their level of risk aversion heavily relies on the precision of the state-action value function estimation when modeling these rare occurrences. Our work proposes to enhance the resilience of RL agents when faced with very rare and risky events by focusing on refining the predictions of the extreme values predicted by the state-action value function distribution. To achieve this, we formulate the extreme values of the state-action value function distribution as parameterized distributions, drawing inspiration from the principles of extreme value theory (EVT). This approach effectively addresses the issue of infrequent occurrence by leveraging EVT-based parameterization. Importantly, we theoretically demonstrate the advantages of employing these parameterized distributions in contrast to other risk-averse algorithms. Our evaluations show that the proposed method outperforms other risk averse RL algorithms on a diverse range of benchmark tasks, each encompassing distinct risk scenarios.