# Automatic Risk Adaptation in Distributional Reinforcement Learning

arXiv: https://arxiv.org/abs/2106.06317 · HTML: https://mlsanity.com/paper/2106.06317/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 13 of 13)

- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Algorithms for CVaR Optimization in MDPs](https://mlsanity.com/paper/1406.3339/) (2014)
- [Safe Reinforcement Learning via Curriculum Induction](https://mlsanity.com/paper/2006.12136/) (2020)
- [Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy](https://mlsanity.com/paper/1911.01546/) (2019)
- [Improving Robustness via Risk Averse Distributional Reinforcement Learning](https://mlsanity.com/paper/2005.00585/) (2020)
- [Risk-Aware Active Inverse Reinforcement Learning](https://mlsanity.com/paper/1901.02161/) (2019)
- [Risk-Conditioned Distributional Soft Actor-Critic for Risk-Sensitive Navigation](https://mlsanity.com/paper/2104.03111/) (2021)

## Built on by (most cited first, 6 of 6)

- [Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning](https://mlsanity.com/paper/2309.14246/) (2023)
- [Train Hard, Fight Easy: Robust Meta Reinforcement Learning](https://mlsanity.com/paper/2301.11147/) (2023)
- [ACReL: Adversarial Conditional value-at-risk Reinforcement Learning](https://mlsanity.com/paper/2109.09470/) (2021)
- [A study of first-passage time minimization via Q-learning in heated gridworlds](https://mlsanity.com/paper/2110.02129/) (2021)
- [DRL-ORA: Distributional Reinforcement Learning with Online Risk Adaption](https://mlsanity.com/paper/2310.05179/) (2023)
- [Risk-Aware Reinforcement Learning for Mobile Manipulation](https://mlsanity.com/paper/2603.04579/) (2026)

## Abstract (arXiv)

The use of Reinforcement Learning (RL) agents in practical applications requires the consideration of suboptimal outcomes, depending on the familiarity of the agent with its environment. This is especially important in safety-critical environments, where errors can lead to high costs or damage. In distributional RL, the risk-sensitivity can be controlled via different distortion measures of the estimated return distribution. However, these distortion functions require an estimate of the risk level, which is difficult to obtain and depends on the current state. In this work, we demonstrate the suboptimality of a static risk level estimation and propose a method to dynamically select risk levels at each environment step. Our method ARA (Automatic Risk Adaptation) estimates the appropriate risk level in both known and unknown environments using a Random Network Distillation error. We show reduced failure rates by up to a factor of 7 and improved generalization performance by up to 14% compared to both risk-aware and risk-agnostic agents in several locomotion environments.