mlsanityData to 14 Oct 2026

Prioritizing Samples in Reinforcement Learning with Reducible Loss

arXiv 2208.10483 ↗First posted 22 Aug 2022cs.LG cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Most reinforcement learning algorithms take advantage of an experience replay buffer to repeatedly train on samples the agent has observed in the past. Not all samples carry the same amount of significance and simply assigning equal importance to each of the samples is a na"ive strategy. In this paper, we propose a method to prioritize samples based on how much we can learn from a sample. We define the learn-ability of a sample as the steady decrease of the training loss associated with this sample over time. We develop an algorithm to prioritize samples with high learn-ability, while assigning lower priority to those that are hard-to-learn, typically caused by noise or stochasticity. We empirically show that our method is more robust than random sampling and also better than just prioritizing with respect to the training loss, i.e. the temporal difference loss, which is used in prioritized experience replay.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 22 Aug 2022.

Builds on

23 of 36 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2018DeepMind Control Suitecited by 956
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  12. 2017Noisy Networks for Explorationcited by 266
11 more
  1. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  2. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  3. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  4. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  5. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41
  6. 2022The Phenomenon of Policy Churncited by 28
  7. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  8. 2019Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Pastcited by 18
  9. 2022Topological Experience Replaycited by 17
  10. 2021Large Batch Experience Replaycited by 14
  11. 2021When does loss-based prioritization fail?cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 7; 2026: 8.

22 papers.

Sort:

By area:

  1. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  2. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  3. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 26
  4. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  5. 2024Prioritized Generative Replaycited by 13
  6. 2024Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNetscited by 8
  7. 2025Selective Learning for Deep Time Series Forecastingcited by 8
  8. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learningcited by 7
  9. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  10. 2023REDUCR: Robust Data Downsampling Using Class Priority Reweightingcited by 3
  11. 2025Off-policy Reinforcement Learning with Model-based Exploration Augmentationcited by 3
  12. 2025Experience Replay with Random Reshufflingcited by 1
10 more
  1. 2026Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVRcited by 1
  2. 2025Unsupervised Active Learning via Natural Feature Progressive Frameworkcited by 0
  3. 2025Bandit Guided Submodular Curriculum for Adaptive Subset Selectioncited by 0
  4. 2026VLM-Guided Experience Replaycited by 0
  5. 2026E2DT: Efficient and Effective Decision Transformer with Experience-Aware Sampling for Robotic Manipulationcited by 0
  6. 2026Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Controlcited by 0
  7. 2026Uncertainty-Driven Replay Memory for Reinforcement Learningcited by 0
  8. 2026Correcting Within-Group Self-Selection Bias in Prioritized Replaycited by 0
  9. 2026Reward Inflation: A Healthy Stimulus for Reinforcement Learningcited by 0
  10. 2026Probabilistic Multi-Source Data Selection via Learnable Categorical Priors: Variance Reduction and Guaranteed Derandomizationcited by 0