# Risk-Aware Transfer in Reinforcement Learning using Successor Features

arXiv: https://arxiv.org/abs/2105.14127 · HTML: https://mlsanity.com/paper/2105.14127/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 62 references that are in the map.
- First posted on arXiv 28 May 2021.

## Builds on (most cited first, 24 of 24)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Deep Reinforcement Learning and the Deadly Triad](https://mlsanity.com/paper/1812.02648/) (2018)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Learning to be Safe: Deep RL with a Safety Critic](https://mlsanity.com/paper/2010.14603/) (2020)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement](https://mlsanity.com/paper/1901.10964/) (2019)
- [Safe Reinforcement Learning via Curriculum Induction](https://mlsanity.com/paper/2006.12136/) (2020)
- [Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret](https://mlsanity.com/paper/2006.13827/) (2020)
- [Risk-Averse Trust Region Optimization for Reward-Volatility Reduction](https://mlsanity.com/paper/1912.03193/) (2019)
- [Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning](https://mlsanity.com/paper/1810.06530/) (2018)
- [Variance Reduction for Reinforcement Learning in Input-Driven Environments](https://mlsanity.com/paper/1807.02264/) (2018)
- [Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning](https://mlsanity.com/paper/2004.10888/) (2020)
- [Entropic Risk Measure in Policy Search](https://mlsanity.com/paper/1906.09090/) (2019)
- [Learning Robust Options](https://mlsanity.com/paper/1802.03236/) (2018)
- [Learning Robust Options by Conditional Value at Risk Optimization](https://mlsanity.com/paper/1905.09191/) (2019)
- [Stochastic Approximation for Risk-aware Markov Decision Processes](https://mlsanity.com/paper/1805.04238/) (2018)
- [Safe Option-Critic: Learning Safety in the Option-Critic Architecture](https://mlsanity.com/paper/1807.08060/) (2018)
- [Variance Penalized On-Policy and Off-Policy Actor-Critic](https://mlsanity.com/paper/2102.01985/) (2021)
- [Probabilistically Safe Policy Transfer](https://mlsanity.com/paper/1705.05394/) (2017)
- [Borrowing From the Future: An Attempt to Address Double Sampling](https://mlsanity.com/paper/1912.00304/) (2019)
- [Situational Awareness by Risk-Conscious Skills](https://mlsanity.com/paper/1610.02847/) (2016)

## Built on by (most cited first, 12 of 12)

- [Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer](https://mlsanity.com/paper/2206.11326/) (2022)
- [A Distributional Analogue to the Successor Representation](https://mlsanity.com/paper/2402.08530/) (2024)
- [Foundations of Multivariate Distributional Reinforcement Learning](https://mlsanity.com/paper/2409.00328/) (2024)
- [Deep Hedging: Continuous Reinforcement Learning for Hedging of General Portfolios across Multiple Risk Aversions](https://mlsanity.com/paper/2207.07467/) (2022)
- [Reinforcement Learning by Guided Safe Exploration](https://mlsanity.com/paper/2307.14316/) (2023)
- [Compositional Planning with Jumpy World Models](https://mlsanity.com/paper/2602.19634/) (2026)
- [Safety-Constrained Policy Transfer with Successor Features](https://mlsanity.com/paper/2211.05361/) (2022)
- [Constructing an Optimal Behavior Basis for the Option Keyboard](https://mlsanity.com/paper/2505.00787/) (2025)
- [Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning](https://mlsanity.com/paper/2310.10818/) (2023)
- [TRAM: Test-Time Risk Adaptation with Mixture of Agents](https://mlsanity.com/paper/2408.08812/) (2024)
- [Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning](https://mlsanity.com/paper/2510.02945/) (2025)
- [Full-Gradient Successor Feature Representations](https://mlsanity.com/paper/2604.00686/) (2026)

## Abstract (arXiv)

Sample efficiency and risk-awareness are central to the development of practical reinforcement learning (RL) for complex decision-making. The former can be addressed by transfer learning and the latter by optimizing some utility function of the return. However, the problem of transferring skills in a risk-aware manner is not well-understood. In this paper, we address the problem of risk-aware policy transfer between tasks in a common domain that differ only in their reward functions, in which risk is measured by the variance of reward streams. Our approach begins by extending the idea of generalized policy improvement to maximize entropic utilities, thus extending policy improvement via dynamic programming to sets of policies and levels of risk-aversion. Next, we extend the idea of successor features (SF), a value function representation that decouples the environment dynamics from the rewards, to capture the variance of returns. Our resulting risk-aware successor features (RaSF) integrate seamlessly within the RL framework, inherit the superior task generalization ability of SFs, and incorporate risk-awareness into the decision-making. Experiments on a discrete navigation domain and control of a simulated robotic arm demonstrate the ability of RaSFs to outperform alternative methods including SFs, when taking the risk of the learned policies into account.