mlsanityData to 7 Oct 2026

Experience Replay with Likelihood-free Importance Weights

arXiv 2006.13169 ↗First posted 23 Jun 2020cs.AI cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

The use of past experiences to accelerate temporal difference (TD) learning of value functions, or experience replay, is a key component in deep reinforcement learning. Prioritization or reweighting of important experiences has shown to improve performance of TD learning algorithms.In this work, we propose to reweight experiences based on their likelihood under the stationary distribution of the current policy. Using the corresponding reweighted TD objective, we implicitly encourage small approximation errors on the value function over frequently encountered states. We use a likelihood-free density ratio estimator over the replay buffer to assign the prioritization weights. We apply the proposed approach empirically on two competitive methods, Soft Actor Critic (SAC) and Twin Delayed Deep Deterministic policy gradient (TD3) -- over a suite of OpenAI gym tasks and achieve superior sample complexity compared to other baseline approaches.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 4; 2023: 14; 2024: 5; 2025: 4; 2026: 5.

42 papers.

Sort:

By area:

  1. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  2. 2021Online Coreset Selection for Rehearsal-based Continual Learningcited by 114
  3. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  4. 2023Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritizationcited by 46
  5. 2022Robust Imitation Learning against Variations in Environment Dynamicscited by 26
  6. 2021Regret Minimization Experience Replay in Off-Policy Reinforcement Learningcited by 25
  7. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  8. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  9. 2022Prioritizing Samples in Reinforcement Learning with Reducible Losscited by 20
  10. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  11. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  12. 2021A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representationcited by 19
30 more
  1. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  2. 2023Curious Replay for Model-based Adaptationcited by 16
  3. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  4. 2023GRASP: A Rehearsal Policy for Efficient Online Continual Learningcited by 13
  5. 2025Data Retrieval with Importance Weights for Few-Shot Imitation Learningcited by 10
  6. 2021Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experiencecited by 8
  7. 2023Decoupled Prioritized Resampling for Offline RLcited by 6
  8. 2023Online Continual Knowledge Learning for Language Modelscited by 6
  9. 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Datacited by 5
  10. 2021Taylor Expansion of Discount Factorscited by 4
  11. 2023Uncertainty-Aware Instance Reweighting for Off-Policy Learningcited by 3
  12. 2023Rethinking Population-assisted Off-policy Reinforcement Learningcited by 3
  13. 2024Revisiting Experience Replayable Conditionscited by 3
  14. 2024ROER: Regularized Optimal Experience Replaycited by 3
  15. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  16. 2021Explaining Off-Policy Actor-Critic From A Bias-Variance Perspectivecited by 2
  17. 2026Self-Organizing Dual-Buffer Adaptive Clustering Experience Replay (SODACER) for Safe Reinforcement Learning in Optimal Controlcited by 2
  18. 2021Model-Augmented Q-learningcited by 1
  19. 2022Off-Policy Reinforcement Learning with Loss Function Weighted by Temporal Difference Errorcited by 1
  20. 2023Improving Representational Continuity via Continued Pretrainingcited by 1
  21. 2025Fitted Q-Evaluation without Bellman Completeness via Occupancy Weightingcited by 1
  22. 2021Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correctioncited by 0
  23. 2022Offline Policy Optimization in RL with Variance Regularizatoncited by 0
  24. 2023Understanding the effect of varying amounts of replay per stepcited by 0
  25. 2024Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiencescited by 0
  26. 2025Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learningcited by 0
  27. 2026VLM-Guided Experience Replaycited by 0
  28. 2026Adaptive TD-Lambda for Cooperative Multi-agent Reinforcement Learningcited by 0
  29. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0
  30. 2026Characterizing Replay Retention Under Dynamics Shift in Model-Based Reinforcement Learningcited by 0