# Fast Task Inference with Variational Intrinsic Successor Features

arXiv: https://arxiv.org/abs/1906.05030 · HTML: https://mlsanity.com/paper/1906.05030/

## Key facts

- Cited by 133 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 30 of 133)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [URLB: Unsupervised Reinforcement Learning Benchmark](https://mlsanity.com/paper/2110.15191/) (2021)
- [Learning One Representation to Optimize All Rewards](https://mlsanity.com/paper/2103.07945/) (2021)
- [APS: Active Pretraining with Successor Features](https://mlsanity.com/paper/2108.13956/) (2021)
- [Pretraining Representations for Data-Efficient Reinforcement Learning](https://mlsanity.com/paper/2106.04799/) (2021)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Does Zero-Shot Reinforcement Learning Exist?](https://mlsanity.com/paper/2209.14935/) (2022)
- [CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00161/) (2022)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning](https://mlsanity.com/paper/2212.08860/) (2022)
- [Temporal Abstraction in Reinforcement Learning with the Successor Representation](https://mlsanity.com/paper/2110.05740/) (2021)
- [Controllability-Aware Unsupervised Skill Discovery](https://mlsanity.com/paper/2302.05103/) (2023)
- [Mask-based Latent Reconstruction for Reinforcement Learning](https://mlsanity.com/paper/2201.12096/) (2022)
- [Learning more skills through optimistic exploration](https://mlsanity.com/paper/2107.14226/) (2021)
- [Interesting Object, Curious Agent: Learning Task-Agnostic Exploration](https://mlsanity.com/paper/2111.13119/) (2021)
- [The Information Geometry of Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2110.02719/) (2021)
- [Choreographer: Learning and Adapting Skills in Imagination](https://mlsanity.com/paper/2211.13350/) (2022)
- [Challenging Common Assumptions in Convex Reinforcement Learning](https://mlsanity.com/paper/2202.01511/) (2022)
- [Human-level Atari 200x faster](https://mlsanity.com/paper/2209.07550/) (2022)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [The Challenges of Exploration for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.11861/) (2022)
- [A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2210.04561/) (2022)
- [PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning](https://mlsanity.com/paper/2102.12560/) (2021)
- [Maximum State Entropy Exploration using Predecessor and Successor Representations](https://mlsanity.com/paper/2306.14808/) (2023)
- [Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer](https://mlsanity.com/paper/2206.11326/) (2022)
- [Discovering a set of policies for the worst case reward](https://mlsanity.com/paper/2102.04323/) (2021)
- [Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning](https://mlsanity.com/paper/2102.13515/) (2021)
- [Dynamic Bottleneck for Robust Self-Supervised Exploration](https://mlsanity.com/paper/2110.10735/) (2021)

## Abstract (arXiv)

It has been established that diverse behaviors spanning the controllable subspace of an Markov decision process can be trained by rewarding a policy for being distinguishable from other policies citepgregor2016variational, eysenbach2018diversity, warde2018unsupervised. However, one limitation of this formulation is generalizing behaviors beyond the finite set being explicitly learned, as is needed for use on subsequent tasks. Successor features citepdayan93improving, barreto2017successor provide an appealing solution to this generalization problem, but require defining the reward function as linear in some grounded feature space. In this paper, we show that these two techniques can be combined, and that each method solves the other's primary limitation. To do so we introduce Variational Intrinsic Successor FeatuRes (VISR), a novel algorithm which learns controllable features that can be leveraged to provide enhanced generalization and fast task inference through the successor feature framework. We empirically validate VISR on the full Atari suite, in a novel setup wherein the rewards are only exposed briefly after a long unsupervised phase. Achieving human-level performance on 14 games and beating all baselines, we believe VISR represents a step towards agents that rapidly learn from limited feedback.