mlsanityData to 7 Oct 2026

Off-Policy Actor-Critic with Shared Experience Replay

arXiv 1909.11583 ↗First posted 25 Sep 2019cs.LG cs.AI stat.MLcited by 39

Not on the map. Why?

Abstract (arXiv)

We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module. To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods. Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable. We provide extensive empirical validation of the proposed solution. We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 17; 2022: 13; 2023: 3; 2024: 5; 2025: 1.

39 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  3. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  4. 2021Bootstrapped Meta-Learningcited by 45
  5. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  6. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  7. 2022Human-level Atari 200x fastercited by 28
  8. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  9. 2023Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selectioncited by 14
  10. 2023Thinker: Learning to Plan and Actcited by 13
  11. 2021A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutionscited by 12
  12. 2022Generalized Data Distribution Iterationcited by 12
27 more
  1. 2021Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPscited by 11
  2. 2022Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENScited by 10
  3. 2021Measuring Progress in Deep Reinforcement Learning Sample Efficiencycited by 8
  4. 2021Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experiencecited by 8
  5. 2022On Realization of Intelligent Decision-Making in the Real World: A Foundation Decision Model Perspectivecited by 8
  6. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
  7. 2021An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learningcited by 7
  8. 2021GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learningcited by 7
  9. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  10. 2022Control of Dual-Sourcing Inventory Systems using Recurrent Neural Networkscited by 6
  11. 2024The Curse of Diversity in Ensemble-Based Explorationcited by 6
  12. 2021Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updatescited by 5
  13. 2024Transductive Off-policy Proximal Policy Optimizationcited by 5
  14. 2022Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approachcited by 3
  15. 2024Learning Diverse Policies with Soft Self-Generated Guidancecited by 3
  16. 2021Behavior-based Neuroevolutionary Training in Reinforcement Learningcited by 2
  17. 2021Explaining Off-Policy Actor-Critic From A Bias-Variance Perspectivecited by 2
  18. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithmscited by 2
  19. 2022Safe and Robust Experience Sharing for Deterministic Policy Gradient Algorithmscited by 2
  20. 2021CASA: Bridging the Gap between Policy Improvement and Policy Evaluation with Conflict Averse Policy Iterationcited by 1
  21. 2022Coupling User Preference with External Rewards to Enable Driver-centered and Resource-aware EV Charging Recommendationcited by 1
  22. 2022Coordinate Ascent for Off-Policy RL with Global Convergence Guaranteescited by 1
  23. 2021Parameter-free Reduction of the Estimation Bias in Deep Reinforcement Learning for Deterministic Policy Gradientscited by 0
  24. 2021Automatic tuning of hyper-parameters of reinforcement learning algorithms using Bayesian optimization with behavioral cloningcited by 0
  25. 2022Non-Markovian policies occupancy measurescited by 0
  26. 2024Off-OAB: Off-Policy Policy Gradient Method with Optimal Action-Dependent Baselinecited by 0
  27. 2025An Arbitration Control for an Ensemble of Diversified DQN variants in Continual Reinforcement Learningcited by 0