mlsanityData to 14 Oct 2026

Reward-Aware Proto-Representations in Reinforcement Learning

arXiv 2505.16217 ↗First posted 22 May 2025cs.LGcited by 4

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In recent years, the successor representation (SR) has attracted increasing attention in reinforcement learning (RL), and it has been used to address some of its key challenges, such as exploration, credit assignment, and generalization. The SR can be seen as representing the underlying credit assignment structure of the environment by implicitly encoding its induced transition dynamics. However, the SR is reward-agnostic. In this paper, we discuss a similar representation that also takes into account the reward dynamics of the problem. We study the default representation (DR), a recently proposed representation with limited theoretical (and empirical) analysis. Here, we lay some of the theoretical foundation underlying the DR in the tabular case by (1) deriving dynamic programming and (2) temporal-difference methods to learn the DR, (3) characterizing the basis for the vector space of the DR, and (4) formally extending the DR to the function approximation case through default features. Empirically, we analyze the benefits of the DR in many of the settings in which the SR has been applied, including (1) reward shaping, (2) option discovery, (3) exploration, and (4) transfer learning. Our results show that, compared to the SR, the DR gives rise to qualitatively different, reward-aware behaviour and quantitatively better performance in several settings.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 32 of its 58 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

32 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models15 Apr
  2. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agent29 Nov
  3. 2024Learning Successor Features the Simple Way29 Oct
  4. 2024A Distributional Analogue to the Successor Representation13 Feb

Other references · 28

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  3. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  4. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  5. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  6. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  7. 2021Learning One Representation to Optimize All Rewardscited by 115
  8. 2018Count-Based Exploration with the Successor Representationcited by 112
  9. 2017A Laplacian Framework for Option Discovery in Reinforcement Learningcited by 110
  10. 2018Universal Successor Features Approximatorscited by 103
  11. 2017Eigenoption Discovery through the Deep Successor Representationcited by 86
  12. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
16 more
  1. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
  2. 2016Deep Successor Reinforcement Learningcited by 72
  3. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  4. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 54
  5. 2021Towards Robust Bisimulation Metric Learningcited by 49
  6. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  7. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42
  8. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 29
  9. 2022On the Generalization of Representations in Reinforcement Learningcited by 28
  10. 2021Towards Better Laplacian Representation in Reinforcement Learning with Generalized Graph Drawingcited by 23
  11. 2021A First-Occupancy Representation for Reinforcement Learningcited by 14
  12. 2021SimSR: Simple Distance-based State Representation for Deep Reinforcement Learningcited by 14
  13. 2023Proper Laplacian Representation Learningcited by 11
  14. 2022Reachability-Aware Laplacian Representation in Reinforcement Learningcited by 10
  15. 2023A State Representation for Diminishing Rewardscited by 3
  16. 2020Goal Kernel Planning: Linearly-Solvable Non-Markovian Policies for Logical Tasks with Goal-Conditioned Optionscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026DROGO: Default Representation Objective via Graph Optimization in Reinforcement Learningcited by 0
  2. 2026SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Controlcited by 0
  3. 2026The Terminal Representation in Reinforcement Learningcited by 0
  4. 2026Mastering Atari 2600 Games with Discovered Optionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learning
  2. 2026Robust Successor Features
  3. 2023Harnessing Discrete Representations For Continual Reinforcement Learning
  4. 2025A Survey of State Representation Learning for Deep Reinforcement Learning
  5. 2024Predictive representations: building blocks of intelligence
  6. 2024Bridging State and History Representations: Understanding Self-Predictive RL
  7. 2025SR-Reward: Taking The Path More Traveled
  8. 2026Can We Really Learn One Representation to Optimize All Rewards?