mlsanityData to 7 Oct 2026

An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay

arXiv 2007.06049 ↗First posted 12 Jul 2020cs.LG stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

Prioritized Experience Replay (PER) is a deep reinforcement learning technique in which agents learn from transitions sampled with non-uniform probability proportionate to their temporal-difference error. We show that any loss function evaluated with non-uniformly sampled data can be transformed into another uniformly sampled loss function with the same expected gradient. Surprisingly, we find in some environments PER can be replaced entirely by this new loss function without impact to empirical performance. Furthermore, this relationship suggests a new branch of improvements to PER by correcting its uniformly sampled loss function equivalent. We demonstrate the effectiveness of our proposed modifications to PER and the equivalent loss function in several MuJoCo and Atari environments.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 9; 2024: 4; 2025: 4; 2026: 13.

41 papers.

Sort:

By area:

  1. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  2. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 46
  3. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  4. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  5. 2022Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfercited by 22
  6. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 20
  7. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  8. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
  9. 2022Actor Prioritized Experience Replaycited by 19
  10. 2021Class-Incremental Experience Replay for Continual Learning under Concept Driftcited by 17
  11. 2022Live in the Moment: Learning Dynamics Model Adapted to Evolving Policycited by 17
  12. 2021Large Batch Experience Replaycited by 14
29 more
  1. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  2. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8
  3. 2023Conversational Tree Search: A New Hybrid Dialog Taskcited by 7
  4. 2023Imitation Learning from Observation through Optimal Transportcited by 7
  5. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learningcited by 5
  6. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learningcited by 5
  7. 2022Introspective Experience Replay: Look Back When Surprisedcited by 3
  8. 2024ROER: Regularized Optimal Experience Replaycited by 3
  9. 2025Spectral Representation-based Reinforcement Learningcited by 3
  10. 2021Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularizationcited by 2
  11. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  12. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  13. 2023Importance Sampling for Stochastic Gradient Descent in Deep Neural Networkscited by 1
  14. 2023Attention Loss Adjusted Prioritized Experience Replaycited by 1
  15. 2023Directly Attention Loss Adjusted Prioritized Experience Replaycited by 1
  16. 2025Experience Replay with Random Reshufflingcited by 1
  17. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  18. 2023Discerning Temporal Difference Learningcited by 0
  19. 2024Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiencescited by 0
  20. 2025SALE-Based Offline Reinforcement Learning with Ensemble Q-Networkscited by 0
  21. 2026D-SPEAR: Dual-Stream Prioritized Experience Adaptive Replay for Stable Reinforcement Learning in Robotic Manipulationcited by 0
  22. 2026NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learningcited by 0
  23. 2026Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entrycited by 0
  24. 2026Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learningcited by 0
  25. 2026Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Controlcited by 0
  26. 2026ProDVI: Programmatic Dynamics Priors for Value Network Initializationcited by 0
  27. 2026Beyond Isolation: Unlocking Reinforcement Learning Component Synergy for Sample-Efficient Continuous Controlcited by 0
  28. 2026Uncertainty-Driven Replay Memory for Reinforcement Learningcited by 0
  29. 2026Correcting Within-Group Self-Selection Bias in Prioritized Replaycited by 0