mlsanityData to 14 Oct 2026

Space-sampled Value Decay: Forgetting Mechanisms for Non-stationary Reinforcement Learning

arXiv 2606.11797 ↗First posted 10 Jun 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Reinforcement Learning agents deployed on physical systems must adapt continually, since degradation and shifting environment conditions change the dynamics (they drift) over time. In the hardest version of this problem, the agent interacts with a single system that might drift at every timestep, leaving no opportunity to revisit past conditions -- a setting we call Single Environment, One-Shot Non-Stationary Reinforcement Learning (SEOS-NSRL). We argue that this setting calls for selective forgetting rather than re-learning, and introduce Space-sampled Value Decay (SsVD), which pulls value estimates of randomly chosen elements of the state space to a baseline value, so that outdated information in non visited regions is discarded. SsVD does not require resetting or change-point detection and plugs into modern off-policy algorithms; we integrate it into Soft Actor Critic and Deep Q-Networks. Across 6 non-stationary environments, SsVD improves upon its direct base algorithms and attains the best mean rank across all. The SsVD mechanism can also induce optimism which we show on hard-exploration tasks, although we investigate the connection here only briefly.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 26 of its 54 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

26 of 54 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2018Experience Replay for Continual Learningcited by 1,095
  7. 2018Exploration by Random Network Distillationcited by 993
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  10. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  11. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  12. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
14 more
  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  2. 2017Deep Exploration via Randomized Value Functionscited by 156
  3. 2019Behaviour Suite for Reinforcement Learningcited by 132
  4. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 108
  5. 2023A Definition of Continual Reinforcement Learningcited by 101
  6. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 85
  7. 2023Maintaining Plasticity in Continual Learning via Regenerative Regularizationcited by 65
  8. 2020Smooth Exploration for Robotic Reinforcement Learningcited by 54
  9. 2023Curious Replay for Model-based Adaptationcited by 16
  10. 2022Task-Agnostic Continual Reinforcement Learning: Gaining Insights and Overcoming Challengescited by 11
  11. 2023A Definition of Non-Stationary Banditscited by 9
  12. 2014Domain-Independent Optimistic Initialization for Reinforcement Learningcited by 8
  13. 2026Chebyshev Policies and the Mountain Car Problem: Reinforcement Learning for Low-Dimensional Control Taskscited by 1
  14. 2026Position: Deployed Reinforcement Learning should be Continualcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Pausing Policy Learning in Non-stationary Reinforcement Learning
  2. 2025Efficient Restarts in Non-Stationary Model-Free Reinforcement Learning
  3. 2024Is Prior-Free Black-Box Non-Stationary Reinforcement Learning Feasible?
  4. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning
  5. 2025Natural Policy Gradient for Average Reward Non-Stationary RL
  6. 2024Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset
  7. 2024Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces
  8. 2026Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities