# EX-DRL: Hedging Against Heavy Losses with EXtreme Distributional Reinforcement Learning

arXiv: https://arxiv.org/abs/2408.12446 · HTML: https://mlsanity.com/paper/2408.12446/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [Adversarial robustness and RL in finance](https://mlsanity.com/deeprl/adversarial-robustness-and-rl-in-finance/)

## Builds on (most cited first, 19 of 19)

- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Deep Hedging](https://mlsanity.com/paper/1802.03042/) (2018)
- [Conservative Offline Distributional Reinforcement Learning](https://mlsanity.com/paper/2107.06106/) (2021)
- [Deep Hedging of Derivatives Using Reinforcement Learning](https://mlsanity.com/paper/2103.16409/) (2021)
- [QLBS: Q-Learner in the Black-Scholes(-Merton) Worlds](https://mlsanity.com/paper/1712.04609/) (2017)
- [Gamma and Vega Hedging Using Deep Distributional Reinforcement Learning](https://mlsanity.com/paper/2205.05614/) (2022)
- [Option Hedging with Risk Averse Reinforcement Learning](https://mlsanity.com/paper/2010.12245/) (2020)
- [Deep Hedging: Continuous Reinforcement Learning for Hedging of General Portfolios across Multiple Risk Aversions](https://mlsanity.com/paper/2207.07467/) (2022)
- [An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient](https://mlsanity.com/paper/2307.08873/) (2023)
- [Understanding the Under-Coverage Bias in Uncertainty Estimation](https://mlsanity.com/paper/2106.05515/) (2021)
- [Hedging Beyond the Mean: A Distributional Reinforcement Learning Perspective for Hedging Portfolios with Structured Products](https://mlsanity.com/paper/2407.10903/) (2024)
- [Bias-Corrected Peaks-Over-Threshold Estimation of the CVaR](https://mlsanity.com/paper/2103.05059/) (2021)
- [Estimation and uncertainty quantification for extreme quantile regions](https://mlsanity.com/paper/1904.08251/) (2019)
- [Equal Risk Pricing and Hedging of Financial Derivatives with Convex Risk Measures](https://mlsanity.com/paper/2002.02876/) (2020)
- [A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning](https://mlsanity.com/paper/2401.02325/) (2024)
- [A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty](https://mlsanity.com/paper/2401.02914/) (2024)
- [Catastrophic-risk-aware reinforcement learning with extreme-value-theory-based policy gradients](https://mlsanity.com/paper/2406.15612/) (2024)

## Abstract (arXiv)

Recent advancements in Distributional Reinforcement Learning (DRL) for modeling loss distributions have shown promise in developing hedging strategies in derivatives markets. A common approach in DRL involves learning the quantiles of loss distributions at specified levels using Quantile Regression (QR). This method is particularly effective in option hedging due to its direct quantile-based risk assessment, such as Value at Risk (VaR) and Conditional Value at Risk (CVaR). However, these risk measures depend on the accurate estimation of extreme quantiles in the loss distribution's tail, which can be imprecise in QR-based DRL due to the rarity and extremity of tail data, as highlighted in the literature. To address this issue, we propose EXtreme DRL (EX-DRL), which enhances extreme quantile prediction by modeling the tail of the loss distribution with a Generalized Pareto Distribution (GPD). This method introduces supplementary data to mitigate the scarcity of extreme quantile observations, thereby improving estimation accuracy through QR. Comprehensive experiments on gamma hedging options demonstrate that EX-DRL improves existing QR-based models by providing more precise estimates of extreme quantiles, thereby improving the computation and reliability of risk metrics for complex financial risk management.