mlsanityData to 7 Oct 2026

Experience Replay Optimization

arXiv 1906.08387 ↗First posted 19 Jun 2019cs.LG stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consider learning a replay policy to optimize the cumulative reward. Replay learning is challenging because the replay memory is noisy and large, and the cumulative reward is unstable. To address these issues, we propose a novel experience replay optimization (ERO) framework which alternately updates two policies: the agent policy, and the replay policy. The agent is updated to maximize the cumulative reward based on the replayed data, while the replay policy is updated to provide the agent with the most useful experiences. The conducted experiments on various continuous control tasks demonstrate the effectiveness of ERO, empirically showing promise in experience replay learning to improve the performance of off-policy reinforcement learning algorithms.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 12; 2023: 11; 2024: 4; 2025: 4; 2026: 3.

47 papers.

Sort:

By area:

  1. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  2. 2022A Survey on Graph Representation Learning Methodscited by 66
  3. 2021DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learningcited by 65
  4. 2021FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Financecited by 57
  5. 2021Deep Reinforcement Learning in Computer Vision: A Comprehensive Surveycited by 42
  6. 2021Deep reinforcement learning in medical imaging: A literature reviewcited by 34
  7. 2021Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environmentscited by 28
  8. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  9. 2022Actor Prioritized Experience Replaycited by 19
  10. 2022AutoShard: Automated Embedding Table Sharding for Recommender Systemscited by 17
  11. 2023Curious Replay for Model-based Adaptationcited by 16
  12. 2022Deep reinforcement learning guided graph neural networks for brain network analysiscited by 12
35 more
  1. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learningcited by 12
  2. 2021Simplifying Deep Reinforcement Learning via Self-Supervisioncited by 9
  3. 2022Analysis of Stochastic Processes through Replay Bufferscited by 9
  4. 2022Towards Automated Imbalanced Learning with Deep Hierarchical Reinforcement Learningcited by 9
  5. 2023Context-aware Domain Adaptation for Time Series Anomaly Detectioncited by 8
  6. 2021Revisiting Prioritized Experience Replay: A Value Perspectivecited by 7
  7. 2021AutoVideo: An Automated Video Action Recognition Systemcited by 6
  8. 2021Sim and Real: Better Togethercited by 6
  9. 2023Pre-train and Search: Efficient Embedding Table Sharding with Pre-trained Neural Cost Modelscited by 5
  10. 2025Uncertainty Prioritized Experience Replaycited by 5
  11. 2022Event Tables for Efficient Experience Replaycited by 4
  12. 2021Off-Policy Correction for Deep Deterministic Policy Gradient Algorithms via Batch Prioritized Experience Replaycited by 3
  13. 2022Neighborhood Mixup Experience Replay: Local Convex Interpolation for Improved Sample Efficiency in Continuous Control Taskscited by 3
  14. 2022Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approachcited by 3
  15. 2023ChemGymRL: An Interactive Framework for Reinforcement Learning for Digital Chemistrycited by 3
  16. 2023High Throughput Training of Deep Surrogates from Large Ensemble Runscited by 3
  17. 2024ROER: Regularized Optimal Experience Replaycited by 3
  18. 2025Reliability-Adjusted Prioritized Experience Replaycited by 3
  19. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learningcited by 3
  20. 2021Explanation-Aware Experience Replay in Rule-Dense Environmentscited by 2
  21. 2021Model-Augmented Q-learningcited by 1
  22. 2022Adaptive Risk-Aware Bidding with Budget Constraint in Display Advertisingcited by 1
  23. 2023Towards Personalized Preprocessing Pipeline Searchcited by 1
  24. 2024CUER: Corrected Uniform Experience Replay for Off-Policy Continuous Deep Reinforcement Learning Algorithmscited by 1
  25. 2021Improving Experience Replay through Modeling of Similar Transitions' Setscited by 0
  26. 2022MEET: A Monte Carlo Exploration-Exploitation Trade-off for Buffer Samplingcited by 0
  27. 2023Interactive System-wise Anomaly Detectioncited by 0
  28. 2023Learning and reusing primitive behaviours to improve Hindsight Experience Replay sample efficiencycited by 0
  29. 2023Enhanced Generalization through Prioritization and Diversity in Self-Imitation Reinforcement Learning over Procedural Environments with Sparse Rewardscited by 0
  30. 2024Frugal Actor-Critic: Sample Efficient Off-Policy Deep Reinforcement Learning Using Unique Experiencescited by 0
  31. 2024Shared-unique Features and Task-aware Prioritized Sampling on Multi-task Reinforcement Learningcited by 0
  32. 2025Enhancing Deep Deterministic Policy Gradients on Continuous Control Tasks with Decoupled Prioritized Experience Replaycited by 0
  33. 2026VLM-Guided Experience Replaycited by 0
  34. 2026Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learningcited by 0
  35. 2026Uncertainty-Driven Replay Memory for Reinforcement Learningcited by 0