mlsanityData to 7 Oct 2026

Combining Behaviors with the Successor Features Keyboard

arXiv 2310.15940 ↗First posted 24 Oct 2023cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

The Option Keyboard (OK) was recently proposed as a method for transferring behavioral knowledge across tasks. OK transfers knowledge by adaptively combining subsets of known behaviors using Successor Features (SFs) and Generalized Policy Improvement (GPI). However, it relies on hand-designed state-features and task encodings which are cumbersome to design for every new environment. In this work, we propose the "Successor Features Keyboard" (SFK), which enables transfer with discovered state-features and task encodings. To enable discovery, we propose the "Categorical Successor Feature Approximator" (CSFA), a novel learning algorithm for estimating SFs while jointly discovering state-features and task encodings. With SFK and CSFA, we achieve the first demonstration of transfer with SFs in a challenging 3D environment where all the necessary representations are discovered. We first compare CSFA against other methods for approximating SFs and show that only CSFA discovers representations compatible with SF&GPI at this scale. We then compare SFK against transfer learning baselines and show that it transfers most quickly to long-horizon tasks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

30 of 42 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2016Progressive Neural Networkscited by 1,332
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  7. 2015Policy Distillationcited by 414
  8. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  9. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  11. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
  12. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
18 more
  1. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 147
  2. 2018Universal Successor Features Approximatorscited by 102
  3. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  4. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  5. 2018Improving Regression Performance with Distributional Lossescited by 66
  6. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 65
  7. 2019COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Explorationcited by 55
  8. 2020Imitating Interactive Intelligencecited by 55
  9. 2020Grounded Language Learning Fast and Slowcited by 54
  10. 2019Environmental drivers of systematicity and generalization in a situated agentcited by 50
  11. 2017Visual Semantic Planning using Deep Successor Representationscited by 45
  12. 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
  13. 2020UneVEn: Universal Value Exploration for Multi-Agent Reinforcement Learningcited by 32
  14. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  15. 2021A New Representation of Successor Features for Transfer across Dissimilar Environmentscited by 13
  16. 2022Zipfian environments for Reinforcement Learningcited by 13
  17. 2023Composing Task Knowledge with Modular Successor Feature Approximatorscited by 7
  18. 2022GrASP: Gradient-Based Affordance Selection for Planningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 5.

12 papers.

  1. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  2. 2024A Distributional Analogue to the Successor Representationcited by 15
  3. 2024Learning Successor Features the Simple Waycited by 10
  4. 2024Predictive representations: building blocks of intelligencecited by 4
  5. 2026Compositional Planning with Jumpy World Modelscited by 4
  6. 2025Constructing an Optimal Behavior Basis for the Option Keyboardcited by 2
  7. 2025NiceWebRL: a Python library for human subject experiments with reinforcement learning environmentscited by 1
  8. 2026Reward-Conditioned Reinforcement Learningcited by 1
  9. 2025A Unified Theory of Compositionality, Modularity, and Interpretability in Markov Decision Processescited by 0
  10. 2026The Laplacian Keyboard: Beyond the Linear Spancited by 0
  11. 2026Hierarchical Successor Representation for Robust Transfercited by 0
  12. 2026Task diversity produces systematic transfer but inhibits continual reinforcement learningcited by 0