# Off-Policy Actor-Critic with Shared Experience Replay

arXiv: https://arxiv.org/abs/1909.11583 · HTML: https://mlsanity.com/paper/1909.11583/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2019.

## Built on by (most cited first, 30 of 39)

- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Online and Offline Reinforcement Learning by Planning with a Learned Model](https://mlsanity.com/paper/2104.06294/) (2021)
- [Muesli: Combining Improvements in Policy Optimization](https://mlsanity.com/paper/2104.06159/) (2021)
- [Bootstrapped Meta-Learning](https://mlsanity.com/paper/2109.04504/) (2021)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [Human-level Atari 200x faster](https://mlsanity.com/paper/2209.07550/) (2022)
- [Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatch](https://mlsanity.com/paper/2111.02997/) (2021)
- [Learnable Behavior Control: Breaking Atari Human World Records via Sample-Efficient Behavior Selection](https://mlsanity.com/paper/2305.05239/) (2023)
- [Thinker: Learning to Plan and Act](https://mlsanity.com/paper/2307.14993/) (2023)
- [A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutions](https://mlsanity.com/paper/2112.04145/) (2021)
- [Deep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs](https://mlsanity.com/paper/2112.09025/) (2021)
- [Generalized Data Distribution Iteration](https://mlsanity.com/paper/2206.03192/) (2022)
- [Social Network Structure Shapes Innovation: Experience-sharing in RL with SAPIENS](https://mlsanity.com/paper/2206.05060/) (2022)
- [Measuring Progress in Deep Reinforcement Learning Sample Efficiency](https://mlsanity.com/paper/2102.04881/) (2021)
- [Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience](https://mlsanity.com/paper/2109.11767/) (2021)
- [On Realization of Intelligent Decision-Making in the Real World: A Foundation Decision Model Perspective](https://mlsanity.com/paper/2212.12669/) (2022)
- [Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Survey](https://mlsanity.com/paper/2411.05614/) (2024)
- [An Entropy Regularization Free Mechanism for Policy-based Reinforcement Learning](https://mlsanity.com/paper/2106.00707/) (2021)
- [GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning](https://mlsanity.com/paper/2106.06232/) (2021)
- [Model-Value Inconsistency as a Signal for Epistemic Uncertainty](https://mlsanity.com/paper/2112.04153/) (2021)
- [Control of Dual-Sourcing Inventory Systems using Recurrent Neural Networks](https://mlsanity.com/paper/2201.06126/) (2022)
- [The Curse of Diversity in Ensemble-Based Exploration](https://mlsanity.com/paper/2405.04342/) (2024)
- [Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updates](https://mlsanity.com/paper/2109.14727/) (2021)
- [Transductive Off-policy Proximal Policy Optimization](https://mlsanity.com/paper/2406.03894/) (2024)
- [Behavior-based Neuroevolutionary Training in Reinforcement Learning](https://mlsanity.com/paper/2105.07960/) (2021)
- [Mitigating Off-Policy Bias in Actor-Critic Methods with One-Step Q-learning: A Novel Correction Approach](https://mlsanity.com/paper/2208.00755/) (2022)
- [Learning Diverse Policies with Soft Self-Generated Guidance](https://mlsanity.com/paper/2402.04539/) (2024)
- [Explaining Off-Policy Actor-Critic From A Bias-Variance Perspective](https://mlsanity.com/paper/2110.02421/) (2021)
- [Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithms](https://mlsanity.com/paper/2202.07496/) (2022)

## Abstract (arXiv)

We investigate the combination of actor-critic reinforcement learning algorithms with uniform large-scale experience replay and propose solutions for two challenges: (a) efficient actor-critic learning with experience replay (b) stability of off-policy learning where agents learn from other agents behaviour. We employ those insights to accelerate hyper-parameter sweeps in which all participating agents run concurrently and share their experience via a common replay module. To this end we analyze the bias-variance tradeoffs in V-trace, a form of importance sampling for actor-critic methods. Based on our analysis, we then argue for mixing experience sampled from replay with on-policy experience, and propose a new trust region scheme that scales effectively to data distributions where V-trace becomes unstable. We provide extensive empirical validation of the proposed solution. We further show the benefits of this setup by demonstrating state-of-the-art data efficiency on Atari among agents trained up until 200M environment frames.