mlsanityData to 14 Oct 2026

A First-Occupancy Representation for Reinforcement Learning

arXiv 2109.13863 ↗First posted 28 Sep 2021cs.LG cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Both animals and artificial agents benefit from state representations that support rapid transfer of learning across tasks and which enable them to efficiently traverse their environments to reach rewarding states. The successor representation (SR), which measures the expected cumulative, discounted state occupancy under a fixed policy, enables efficient transfer to different reward structures in an otherwise constant Markovian environment and has been hypothesized to underlie aspects of biological behavior and neural activity. However, in the real world, rewards may move or only be available for consumption once, may shift location, or agents may simply aim to reach goal states as rapidly as possible without the constraint of artificially imposed task horizons. In such cases, the most behaviorally-relevant representation would carry information about when the agent was likely to first reach states of interest, rather than how often it should expect to visit them over a potentially infinite time span. To reflect such demands, we introduce the first-occupancy representation (FR), which measures the expected temporal discount to the first time a state is accessed. We demonstrate that the FR facilitates exploration, the selection of efficient paths to desired states, allows the agent, under certain conditions, to plan provably optimal trajectories defined by a sequence of subgoals, and induces similar behavior to animals avoiding threatening stimuli.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 28 Sep 2021.

Builds on

23 of 46 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2016OpenAI Gymcited by 1,736
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  7. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  8. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  9. 2016Variational Intrinsic Controlcited by 267
  10. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  11. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  12. 2018Count-Based Exploration with the Successor Representationcited by 112
11 more
  1. 2021APS: Active Pretraining with Successor Featurescited by 112
  2. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  3. 2016Deep Successor Reinforcement Learningcited by 72
  4. 2017Environment-Independent Task Specifications via GLTLcited by 67
  5. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  6. 2019Reinforcement Learning with Non-Markovian Rewardscited by 45
  7. 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
  8. 2021Discovering a set of policies for the worst case rewardcited by 21
  9. 2020Universal Successor Features for Transfer Reinforcement Learningcited by 20
  10. 2019A neurally plausible model learns successor representations in partially observable environmentscited by 12
  11. 2019Better transfer learning with inferred successor mapscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 4; 2024: 3; 2025: 2; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  2. 2023Maximum State Entropy Exploration using Predecessor and Successor Representationscited by 23
  3. 2021Towards an Understanding of Default Policies in Multitask Policy Optimizationcited by 8
  4. 2023Successor-Predecessor Intrinsic Explorationcited by 6
  5. 2024Predictive representations: building blocks of intelligencecited by 4
  6. 2025Reward-Aware Proto-Representations in Reinforcement Learningcited by 4
  7. 2023A State Representation for Diminishing Rewardscited by 3
  8. 2025SR-Reward: Taking The Path More Traveledcited by 3
  9. 2022Safety-Constrained Policy Transfer with Successor Featurescited by 2
  10. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  11. 2024Learning Hidden Subgoals under Temporal Ordering Constraints in Reinforcement Learningcited by 1
  12. 2022Minimum Description Length Controlcited by 0
2 more
  1. 2022Transfer RL via the Undo Maps Formalismcited by 0
  2. 2026DROGO: Default Representation Objective via Graph Optimization in Reinforcement Learningcited by 0