mlsanityData to 7 Oct 2026

Distributed Prioritized Experience Replay

arXiv 1803.00933 ↗First posted 2 Mar 2018cs.LGcited by 260

Not on the map. Why?

Abstract (arXiv)

We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.

Key facts. Cited by 260 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 3; 2020: 2; 2021: 65; 2022: 55; 2023: 47; 2024: 33; 2025: 33; 2026: 21.

260 papers.

Sort:

By area:

  1. 2023End-to-end Autonomous Driving: Challenges and Frontierscited by 485
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  3. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 79
  4. 2021Off-Belief Learningcited by 72
  5. 2024Simplifying Deep Temporal Difference Learningcited by 71
  6. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  7. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 59
  8. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  9. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  10. 2021Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Explorationcited by 49
  11. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  12. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
38 more
  1. 2021CompilerGym: Robust, Performant Compiler Optimization Environments for AI Researchcited by 41
  2. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  3. 2021Podracer architectures for scalable Reinforcement Learningcited by 37
  4. 2019Positive-Unlabeled Reward Learningcited by 36
  5. 2021Reinforcement Learning for Ridesharing: An Extended Surveycited by 35
  6. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  7. 2021Learning Selective Communication for Multi-Agent Path Findingcited by 32
  8. 2021Distributed Heuristic Multi-Agent Path Finding with Communicationcited by 31
  9. 2022K-level Reasoning for Zero-Shot Coordination in Hanabicited by 30
  10. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  11. 2022Human-level Atari 200x fastercited by 28
  12. 2021Reverb: A Framework For Experience Replaycited by 27
  13. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  14. 2021A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learningcited by 27
  15. 2022MuZero with Self-competition for Rate Control in VP9 Video Compressioncited by 27
  16. 2022Honor of Kings Arena: an Environment for Generalization in Competitive Reinforcement Learningcited by 27
  17. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  18. 2021Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Studycited by 24
  19. 2022Continuous MDP Homomorphisms and Homomorphic Policy Gradientcited by 24
  20. 2021Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximationcited by 23
  21. 2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 23
  22. 2022Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021cited by 23
  23. 2022Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policiescited by 22
  24. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  25. 2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
  26. 2021Efficient Continuous Control with Double Actors and Regularized Criticscited by 20
  27. 2021When should agents explore?cited by 20
  28. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  29. 2023MAC-PO: Multi-Agent Experience Replay via Collective Priority Optimizationcited by 20
  30. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  31. 2024Small Scale Data-Free Knowledge Distillationcited by 20
  32. 2022Actor Prioritized Experience Replaycited by 19
  33. 2021Large Batch Simulation for Deep Reinforcement Learningcited by 18
  34. 2021Action Set Based Policy Optimization for Safe Power Grid Managementcited by 18
  35. 2021Scalable Online Planning via Reinforcement Learning Fine-Tuningcited by 18
  36. 2022Learning Deformable Object Manipulation from Expert Demonstrationscited by 18
  37. 2024SAPG: Split and Aggregate Policy Gradientscited by 17
  38. 2023Curious Replay for Model-based Adaptationcited by 16

50 most cited of 260 All 260 (JSON)