mlsanityData to 7 Oct 2026

Revisiting Experience Replayable Conditions

arXiv 2402.10374 ↗First posted 15 Feb 2024cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Experience replay (ER) used in (deep) reinforcement learning is considered to be applicable only to off-policy algorithms. However, there have been some cases in which ER has been applied for on-policy algorithms, suggesting that off-policyness might be a sufficient condition for applying ER. This paper reconsiders more strict "experience replayable conditions" (ERC) and proposes the way of modifying the existing algorithms to satisfy ERC. In light of this, it is postulated that the instability of policy improvements represents a pivotal factor in ERC. The instability factors are revealed from the viewpoint of metric learning as i) repulsive forces from negative samples and ii) replays of inappropriate experiences. Accordingly, the corresponding stabilization tricks are derived. As a result, it is confirmed through numerical simulations that the proposed stabilization tricks make ER applicable to an advantage actor-critic, an on-policy algorithm. Moreover, its learning performance is comparable to that of a soft actor-critic, a state-of-the-art off-policy algorithm.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 47 of its 70 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

47 of 70 references.

References · 47

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  6. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  12. 2015Prioritized Experience Replaycited by 1,070
35 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  2. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  3. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  4. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  5. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  6. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  7. 2014Learning Fine-grained Image Similarity with Deep Rankingcited by 189
  8. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  9. 2020Revisiting Fundamentals of Experience Replaycited by 171
  10. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 114
  11. 2021Recent Advances in Reinforcement Learning in Financecited by 102
  12. 2020Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learningcited by 99
  13. 2020Hard negative examples are hard, but usefulcited by 82
  14. 2022MyoSuite -- A contact-rich simulation suite for musculoskeletal motor controlcited by 80
  15. 2018Remember and Forget for Experience Replaycited by 42
  16. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  17. 2017Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learningcited by 40
  18. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  19. 2020SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft IIcited by 22
  20. 2022L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learningcited by 22
  21. 2022Actor Prioritized Experience Replaycited by 19
  22. 2021Squareplus: A Softplus-Like Algebraic Rectifiercited by 18
  23. 2019Generalized Off-Policy Actor-Criticcited by 16
  24. 2018Fast deep reinforcement learning using online adjustments from the pastcited by 14
  25. 2019Creating Pro-Level AI for a Real-Time Fighting Game Using Deep Reinforcement Learningcited by 14
  26. 2021A unified view of likelihood ratio and reparameterization gradientscited by 12
  27. 2022AdaTerm: Adaptive T-Distribution Estimated Robust Moments for Noise-Robust Stochastic Gradient Optimizationcited by 10
  28. 2021Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experiencecited by 8
  29. 2023PyFlyt -- UAV Simulation Environments for Reinforcement Learning Researchcited by 7
  30. 2021Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimizationcited by 6
  31. 2018Planning with a Receding Horizon for Manipulation in Clutter using a Learned Value Functioncited by 5
  32. 2022Consolidated Adaptive T-soft Update for Deep Reinforcement Learningcited by 5
  33. 2022Proximal Policy Optimization with Adaptive Threshold for Symmetric Relative Density Ratiocited by 5
  34. 2023Intentionally-underestimated Value Function at Terminal State for Temporal-difference Learning with Mis-designed Rewardcited by 5
  35. 2023Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraintcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Investigating the Interplay of Prioritized Replay and Generalizationcited by 5
  2. 2025CubeDAgger: Interactive Imitation Learning for Dynamic Systems with Efficient yet Low-risk Interactioncited by 0
  3. 2025ES-C51: Expected Sarsa Based C51 Distributional Reinforcement Learning Algorithmcited by 0