# Risk-Averse Trust Region Optimization for Reward-Volatility Reduction

arXiv: https://arxiv.org/abs/1912.03193 · HTML: https://mlsanity.com/paper/1912.03193/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Dec 2019.

## Built on by (most cited first, 30 of 54)

- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [AI in Finance: Challenges, Techniques and Opportunities](https://mlsanity.com/paper/2107.09051/) (2021)
- [Constrained Update Projection Approach to Safe Policy Optimization](https://mlsanity.com/paper/2209.07089/) (2022)
- [Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting](https://mlsanity.com/paper/2206.04038/) (2022)
- [HIST: A Graph-based Framework for Stock Trend Forecasting via Mining Concept-Oriented Shared Information](https://mlsanity.com/paper/2110.13716/) (2021)
- [Learning Multiple Stock Trading Patterns with Temporal Routing Adaptor and Optimal Transport](https://mlsanity.com/paper/2106.12950/) (2021)
- [Challenging Common Assumptions in Convex Reinforcement Learning](https://mlsanity.com/paper/2202.01511/) (2022)
- [Reinforcement Learning for Quantitative Trading](https://mlsanity.com/paper/2109.13851/) (2021)
- [CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learning](https://mlsanity.com/paper/2202.07565/) (2022)
- [DeepScalper: A Risk-Aware Reinforcement Learning Framework to Capture Fleeting Intraday Trading Opportunities](https://mlsanity.com/paper/2201.09058/) (2021)
- [FECAM: Frequency Enhanced Channel Attention Mechanism for Time Series Forecasting](https://mlsanity.com/paper/2212.01209/) (2022)
- [A Game-Theoretic Framework for Managing Risk in Multi-Agent Systems](https://mlsanity.com/paper/2205.15434/) (2022)
- [Risk-Aware Transfer in Reinforcement Learning using Successor Features](https://mlsanity.com/paper/2105.14127/) (2021)
- [An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient](https://mlsanity.com/paper/2307.08873/) (2023)
- [The Evolution of Reinforcement Learning in Quantitative Finance: A Survey](https://mlsanity.com/paper/2408.10932/) (2024)
- [Variance Penalized On-Policy and Off-Policy Actor-Critic](https://mlsanity.com/paper/2102.01985/) (2021)
- [Diffusion Variational Autoencoder for Tackling Stochasticity in Multi-Step Regression Stock Price Prediction](https://mlsanity.com/paper/2309.00073/) (2023)
- [Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples](https://mlsanity.com/paper/2310.07747/) (2023)
- [Multi-relational Graph Diffusion Neural Network with Parallel Retention for Stock Trends Classification](https://mlsanity.com/paper/2401.05430/) (2024)
- [DiffsFormer: A Diffusion Transformer on Stock Factor Augmentation](https://mlsanity.com/paper/2402.06656/) (2024)
- [What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorization](https://mlsanity.com/paper/2207.05161/) (2022)
- [TRC: Trust Region Conditional Value at Risk for Safe Reinforcement Learning](https://mlsanity.com/paper/2312.00344/) (2023)
- [Learning Equilibria in Adversarial Team Markov Games: A Nonconvex-Hidden-Concave Min-Max Optimization Problem](https://mlsanity.com/paper/2410.05673/) (2024)
- [Reinforcement Learning with Expert Trajectory For Quantitative Trading](https://mlsanity.com/paper/2105.03844/) (2021)
- [Temporal-Relational Hypergraph Tri-Attention Networks for Stock Trend Prediction](https://mlsanity.com/paper/2107.14033/) (2021)
- [A unified algorithm framework for mean-variance optimization in discounted Markov decision processes](https://mlsanity.com/paper/2201.05737/) (2022)
- [ARLO: A Framework for Automated Reinforcement Learning](https://mlsanity.com/paper/2205.10416/) (2022)
- [Stock Broad-Index Trend Patterns Learning via Domain Knowledge Informed Generative Network](https://mlsanity.com/paper/2302.14164/) (2023)
- [Mlinear: Rethink the Linear Model for Time-series Forecasting](https://mlsanity.com/paper/2305.04800/) (2023)
- [Hedging of Financial Derivative Contracts via Monte Carlo Tree Search](https://mlsanity.com/paper/2102.06274/) (2021)

## Abstract (arXiv)

In real-world decision-making problems, for instance in the fields of finance, robotics or autonomous driving, keeping uncertainty under control is as important as maximizing expected returns. Risk aversion has been addressed in the reinforcement learning literature through risk measures related to the variance of returns. However, in many cases, the risk is measured not only on a long-term perspective, but also on the step-wise rewards (e.g., in trading, to ensure the stability of the investment bank, it is essential to monitor the risk of portfolio positions on a daily basis). In this paper, we define a novel measure of risk, which we call reward volatility, consisting of the variance of the rewards under the state-occupancy measure. We show that the reward volatility bounds the return variance so that reducing the former also constrains the latter. We derive a policy gradient theorem with a new objective function that exploits the mean-volatility relationship, and develop an actor-only algorithm. Furthermore, thanks to the linearity of the Bellman equations defined under the new objective function, it is possible to adapt the well-known policy gradient algorithms with monotonic improvement guarantees such as TRPO in a risk-averse manner. Finally, we test the proposed approach in two simulated financial environments.