mlsanityData to 7 Oct 2026

Hierarchical Successor Representation for Robust Transfer

arXiv 2602.12753 ↗First posted 13 Feb 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The successor representation (SR) provides a powerful framework for decoupling predictive dynamics from rewards, enabling rapid generalisation across reward configurations. However, the classical SR is limited by its inherent policy dependence: policies change due to ongoing learning, environmental non-stationarities, and changes in task demands, making established predictive representations obsolete. Furthermore, in topologically complex environments, SRs suffer from spectral diffusion, leading to dense and overlapping features that scale poorly. Here we propose the Hierarchical Successor Representation (HSR) for overcoming these limitations. By incorporating temporal abstractions into the construction of predictive representations, HSR learns stable state features which are robust to task-induced policy changes. Applying non-negative matrix factorisation (NMF) to the HSR yields a sparse, low-rank state representation that facilitates highly sample-efficient transfer to novel tasks in multi-compartmental environments. Further analysis reveals that HSR-NMF discovers interpretable topological structures, providing a policy-agnostic hierarchical map that effectively bridges model-free optimality and model-based flexibility. Beyond providing a useful basis for task-transfer, we show that HSR's temporally extended predictive structure can also be leveraged to drive efficient exploration, effectively scaling to large, procedurally generated environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 45 references that are in the map. First posted on arXiv 13 Feb 2026.

Builds on

24 of 45 references.

References · 24

  1. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2020Reinforcement Learning with Augmented Datacited by 452
  6. 2016The Option-Critic Architecturecited by 433
  7. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  8. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  9. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  11. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  12. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
12 more
  1. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  2. 2018Generalization and Regularization in DQNcited by 117
  3. 2018Count-Based Exploration with the Successor Representationcited by 112
  4. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 108
  5. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  6. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  7. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  8. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 52
  9. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  10. 2019Discovering Options for Exploration by Minimizing Cover Timecited by 30
  11. 2023Combining Behaviors with the Successor Features Keyboardcited by 12
  12. 2023Successor-Predecessor Intrinsic Explorationcited by 6

Built on by

Not cited yet.