A State Representation for Diminishing Rewards
Not on the map. Why?
Abstract (arXiv)
A common setting in multitask reinforcement learning (RL) demands that an agent rapidly adapt to various stationary reward functions randomly sampled from a fixed distribution. In such situations, the successor representation (SR) is a popular framework which supports rapid policy evaluation by decoupling a policy's expected discounted, cumulative state occupancies from a specific reward function. However, in the natural world, sequential tasks are rarely independent, and instead reflect shifting priorities based on the availability and subjective perception of rewarding stimuli. Reflecting this disjunction, in this paper we study the phenomenon of diminishing marginal utility and introduce a novel state representation, the λ representation (λR) which, surprisingly, is required for policy evaluation in this setting and which generalizes the SR as well as several other state representations from the literature. We establish the λR's formal properties and examine its normative advantages in the context of machine learning, as well as its usefulness for studying natural behaviors, particularly foraging.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 20 of its 43 references that are in the map. First posted on arXiv 7 Sep 2023.
Builds on
20 of 43 references.
References · 20
- 2016Layer Normalizationcited by 5,669
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2016Successor Features for Transfer in Reinforcement Learningcited by 340
- 2019Behaviour Suite for Reinforcement Learningcited by 132
- 2021Learning One Representation to Optimize All Rewardscited by 115
- 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
- 2021Reward is enough for convex MDPscited by 84
- 2022Does Zero-Shot Reinforcement Learning Exist?cited by 83
- 2018The Description Length of Deep Learning Modelscited by 72
- 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
8 more
- 2019Reinforcement Learning with Non-Markovian Rewardscited by 45
- 2020Efficient Wasserstein Natural Gradients for Reinforcement Learningcited by 23
- 2023Submodular Reinforcement Learningcited by 23
- 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 20
- 2021A First-Occupancy Representation for Reinforcement Learningcited by 14
- 2020Planning with Submodular Objective Functionscited by 5
- 2022Neuro-Nav: A Library for Neurally-Plausible Reinforcement Learningcited by 3
- 2022Adaptive patch foraging in deep reinforcement learning agentscited by 3
Built on by
Citing papers per year: 2023: 1; 2024: 0; 2025: 1; 2026: 1.
3 papers.