# Stacked Universal Successor Feature Approximators for Safety in Reinforcement Learning

arXiv: https://arxiv.org/abs/2409.04641 · HTML: https://mlsanity.com/paper/2409.04641/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safe and constrained RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 6 Sep 2024.

## Placed in

- Deep reinforcement learning methods / [Safe and constrained RL](https://mlsanity.com/deeprl/safe-and-constrained-rl/)

## Builds on (most cited first, 12 of 12)

- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement](https://mlsanity.com/paper/1901.10964/) (2019)
- [Expected Eligibility Traces](https://mlsanity.com/paper/2007.01839/) (2020)
- [Run Time Assurance for Safety-Critical Systems: An Introduction to Safety Filtering Approaches for Complex Control Systems](https://mlsanity.com/paper/2110.03506/) (2021)
- [Universal Successor Features for Transfer Reinforcement Learning](https://mlsanity.com/paper/2001.04025/) (2020)
- [Deep Reinforcement Learning for Autonomous Spacecraft Inspection using Illumination](https://mlsanity.com/paper/2308.02743/) (2023)
- [Ablation Study of How Run Time Assurance Impacts the Training and Performance of Reinforcement Learning Agents](https://mlsanity.com/paper/2207.04117/) (2022)
- [Run Time Assurance for Autonomous Spacecraft Inspection](https://mlsanity.com/paper/2302.02885/) (2023)
- [Safety-Constrained Policy Transfer with Successor Features](https://mlsanity.com/paper/2211.05361/) (2022)
- [Multi-Task Reinforcement Learning in Continuous Control with Successor Feature-Based Concurrent Composition](https://mlsanity.com/paper/2303.13935/) (2023)

## Abstract (arXiv)

Real-world problems often involve complex objective structures that resist distillation into reinforcement learning environments with a single objective. Operation costs must be balanced with multi-dimensional task performance and end-states' effects on future availability, all while ensuring safety for other agents in the environment and the reinforcement learning agent itself. System redundancy through secondary backup controllers has proven to be an effective method to ensure safety in real-world applications where the risk of violating constraints is extremely high. In this work, we investigate the utility of a stacked, continuous-control variation of universal successor feature approximation (USFA) adapted for soft actor-critic (SAC) and coupled with a suite of secondary safety controllers, which we call stacked USFA for safety (SUSFAS). Our method improves performance on secondary objectives compared to SAC baselines using an intervening secondary controller such as a runtime assurance (RTA) controller.