mlsanityData to 7 Oct 2026

A Distributional Analogue to the Successor Representation

arXiv 2402.08530 ↗First posted 13 Feb 2024cs.LG cs.AI stat.MLcited by 15

Not on the map. Why?

Abstract (arXiv)

This paper contributes a new approach for distributional reinforcement learning which elucidates a clean separation of transition structure and reward in the learning process. Analogous to how the successor representation (SR) describes the expected consequences of behaving according to a given policy, our distributional successor measure (SM) describes the distributional consequences of this behaviour. We formulate the distributional SM as a distribution over distributions and provide theory connecting it with distributional and model-based reinforcement learning. Moreover, we propose an algorithm that learns the distributional SM from data by minimizing a two-level maximum mean discrepancy. Key to our method are a number of algorithmic techniques that are independently valuable for learning generative models of state. As an illustration of the usefulness of the distributional SM, we show that it enables zero-shot risk-sensitive policy evaluation in a way that was not previously possible.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 42 of its 75 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

42 of 75 references.

References · 42

  1. 2018Demystifying MMD GANscited by 1,268
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  3. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  4. 2018Invertible Residual Networkscited by 364
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  7. 2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 251
  8. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  9. 2017The Cramer Distance as a Solution to Biased Wasserstein Gradientscited by 116
  10. 2018Count-Based Exploration with the Successor Representationcited by 112
  11. 2021Learning One Representation to Optimize All Rewardscited by 110
  12. 2018Universal Successor Features Approximatorscited by 102
30 more
  1. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  2. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  3. 2019Fully Parameterized Quantile Function for Distributional Reinforcement Learningcited by 87
  4. 2017Eigenoption Discovery through the Deep Successor Representationcited by 85
  5. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 79
  6. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 67
  7. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  8. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 52
  9. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  10. 2022Investigating Compounding Prediction Errors in Learned Dynamics Modelscited by 47
  11. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 43
  12. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  13. 2019Discovering Options for Exploration by Minimizing Cover Timecited by 30
  14. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  15. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  16. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 27
  17. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  18. 2021Distributional Reinforcement Learning for Multi-Dimensional Reward Functionscited by 21
  19. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  20. 2020Cautious Reinforcement Learning via Distributional Risk in the Dual Domaincited by 18
  21. 2018Distributional Multivariate Policy Evaluation and Exploration with the Bellman GANcited by 16
  22. 2019A neurally plausible model learns successor representations in partially observable environmentscited by 12
  23. 2021Risk-Aware Transfer in Reinforcement Learning using Successor Featurescited by 12
  24. 2023Combining Behaviors with the Successor Features Keyboardcited by 12
  25. 2022Generalised Policy Improvement with Geometric Policy Compositioncited by 11
  26. 2023Bootstrapped Representations in Reinforcement Learningcited by 11
  27. 2021Safe Distributional Reinforcement Learningcited by 10
  28. 2022A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspacescited by 7
  29. 2023One-Step Distributional Reinforcement Learningcited by 5
  30. 2020A Distributional Analysis of Sampling-Based Reinforcement Learning Algorithmscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025Temporal Difference Flowscited by 20
  2. 2024Foundations of Multivariate Distributional Reinforcement Learningcited by 16
  3. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  4. 2024Distributional Successor Features Enable Zero-Shot Policy Optimizationcited by 8
  5. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  6. 2025A Finite Sample Analysis of Distributional TD Learning with Linear Function Approximationcited by 5
  7. 2024Video Occupancy Modelscited by 4
  8. 2026Compositional Planning with Jumpy World Modelscited by 4
  9. 2025Reward-Aware Proto-Representations in Reinforcement Learningcited by 3
  10. 2025Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learningcited by 2
  11. 2026DROGO: Default Representation Objective via Graph Optimization in Reinforcement Learningcited by 0
  12. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 0
3 more
  1. 2026QnRL: Quantum-Native Reinforcement Learningcited by 0
  2. 2026Compositional Behavioral Semantics for State Abstraction in Reinforcement Learningcited by 0
  3. 2026It's a matter of timescale: non-linear utility in successor features and multi-objective planning and learningcited by 0