mlsanityData to 14 Oct 2026

Sample Efficient Experience Replay in Non-stationary Environments

arXiv 2509.15032 ↗First posted 18 Sep 2025cs.LG cs.AI cs.NIcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Reinforcement learning (RL) in non-stationary environments is challenging, as changing dynamics and rewards quickly make past experiences outdated. Traditional experience replay (ER) methods, especially those using TD-error prioritization, struggle to distinguish between changes caused by the agent's policy and those from the environment, resulting in inefficient learning under dynamic conditions. To address this challenge, we propose the Discrepancy of Environment Dynamics (DoE), a metric that isolates the effects of environment shifts on value functions. Building on this, we introduce Discrepancy of Environment Prioritized Experience Replay (DEER), an adaptive ER framework that prioritizes transitions based on both policy updates and environmental changes. DEER uses a binary classifier to detect environment changes and applies distinct prioritization strategies before and after each shift, enabling more sample-efficient learning. Experiments on four non-stationary benchmarks demonstrate that DEER further improves the performance of off-policy algorithms by 11.54 percent compared to the best-performing state-of-the-art ER methods.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 29 of its 38 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

29 of 38 references.

References · 29

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
  5. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  6. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 176
  7. 2017A Deeper Look at Experience Replaycited by 105
  8. 2023Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networkscited by 95
  9. 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
  10. 2020A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environmentscited by 62
  11. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 62
  12. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
17 more
  1. 2018Remember and Forget for Experience Replaycited by 43
  2. 2023Optimal Resource Allocation for U-Shaped Parallel Split Learningcited by 33
  3. 2025HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 31
  4. 2024FedAC: An Adaptive Clustered Federated Learning Framework for Heterogeneous Datacited by 25
  5. 2025LEO-Split: A Semi-Supervised Split Learning Framework over LEO Satellite Networkscited by 22
  6. 2019Prioritized Sequence Experience Replaycited by 20
  7. 2022Actor Prioritized Experience Replaycited by 20
  8. 2024IC3M: In-Car Multimodal Multi-object Monitoring for Abnormal Status of Both Driver and Passengerscited by 20
  9. 2024MERIT: Multimodal Wearable Vital Sign Waveform Monitoringcited by 16
  10. 2025LCFed: An Efficient Clustered Federated Learning Framework for Heterogeneous Datacited by 15
  11. 2022Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewardscited by 11
  12. 2025Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspectivecited by 10
  13. 2025Dynamic Uncertainty-aware Multimodal Fusion for Outdoor Health Monitoringcited by 10
  14. 2025Robust Deep Reinforcement Learning in Robotics via Adaptive Gradient-Masked Adversarial Attackscited by 7
  15. 2023Replay Buffer with Local Forgetting for Adapting to Local Environment Changes in Deep Model-Based Reinforcement Learningcited by 6
  16. 2025SL-ACC: A Communication-Efficient Split Learning Framework with Adaptive Channel-wise Compressioncited by 6
  17. 2021Optimizing ROC Curves with a Sort-Based Surrogate Loss Function for Binary Classification and Changepoint Detectioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systemscited by 8
  2. 2026Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learningcited by 4
  3. 2026Cognitively-Grounded On-Device Runtime Learning for Ground Robots in Unknown Physical Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Variance Reduction via Resampling and Experience Replay
  2. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learning
  3. 2025Reliability-Adjusted Prioritized Experience Replay
  4. 2026Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learning
  5. 2024Investigating the Interplay of Prioritized Replay and Generalization
  6. 2024Prioritized Generative Replay
  7. 2024ROER: Regularized Optimal Experience Replay
  8. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learning