mlsanityData to 7 Oct 2026

Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning

arXiv 1702.08887 ↗First posted 28 Feb 2017cs.AI cs.LG cs.MAcited by 114

Not on the map. Why?

Abstract (arXiv)

Many real-world problems, such as network packet routing and urban traffic control, are naturally modeled as multi-agent reinforcement learning (RL) problems. However, existing multi-agent RL methods typically scale poorly in the problem size. Therefore, a key challenge is to translate the success of deep learning on single-agent RL to the multi-agent setting. A major stumbling block is that independent Q-learning, the most popular multi-agent RL method, introduces nonstationarity that makes it incompatible with the experience replay memory on which deep Q-learning relies. This paper proposes two methods that address this problem: 1) using a multi-agent variant of importance sampling to naturally decay obsolete data and 2) conditioning each agent's value function on a fingerprint that disambiguates the age of the data sampled from the replay memory. Results on a challenging decentralised variant of StarCraft unit micromanagement confirm that these methods enable the successful combination of experience replay with multi-agent RL.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Feb 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 37; 2022: 23; 2023: 19; 2024: 12; 2025: 14; 2026: 8.

114 papers.

Sort:

By area:

  1. 2022A Survey of Multi-Agent Deep Reinforcement Learning with Communicationcited by 103
  2. 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 97
  3. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 79
  4. 2021Replay in Deep Learning: Current Approaches and Missing Biological Elementscited by 71
  5. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
  6. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  7. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
  8. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  9. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  10. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 30
  11. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  12. 2021Social Coordination and Altruism in Autonomous Drivingcited by 28
38 more
  1. 2023A Review of Cooperation in Multi-agent Learningcited by 27
  2. 2021Influencing Towards Stable Multi-Agent Interactionscited by 24
  3. 2022Scalable Multi-Agent Model-Based Reinforcement Learningcited by 24
  4. 2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 23
  5. 2022Solving Continuous Control via Q-learningcited by 21
  6. 2021Collaborative Visual Navigationcited by 18
  7. 2021Class-Incremental Experience Replay for Continual Learning under Concept Driftcited by 17
  8. 2021Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Trainingcited by 16
  9. 2021Self-Organized Polynomial-Time Coordination Graphscited by 14
  10. 2022Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teamingcited by 14
  11. 2021HAMMER: Multi-Level Coordination of Reinforcement Learning Agents via Learned Messagingcited by 12
  12. 2022Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradientscited by 12
  13. 2022MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learningcited by 12
  14. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learningcited by 12
  15. 2023Dealing With Non-stationarity in Decentralized Cooperative Multi-Agent Deep Reinforcement Learning via Multi-Timescale Learningcited by 12
  16. 2021Spatial Intention Maps for Multi-Agent Mobile Manipulationcited by 11
  17. 2022Distributed Reinforcement Learning for Robot Teams: A Reviewcited by 11
  18. 2023From Explicit Communication to Tacit Cooperation:A Novel Paradigm for Cooperative MARLcited by 11
  19. 2021Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learningcited by 10
  20. 2022EMVLight: a Multi-agent Reinforcement Learning Framework for an Emergency Vehicle Decentralized Routing and Traffic Signal Control Systemcited by 9
  21. 2021Emergence of Theory of Mind Collaboration in Multiagent Systemscited by 8
  22. 2021A Decentralized Reinforcement Learning Framework for Efficient Passage of Emergency Vehiclescited by 8
  23. 2022Collaborative Target Search with a Visual Drone Swarm: An Adaptive Curriculum Embedded Multistage Reinforcement Learning Approachcited by 8
  24. 2023Deep reinforcement learning for machine scheduling: Methodology, the state-of-the-art, and future directionscited by 8
  25. 2019Independent Generative Adversarial Self-Imitation Learning in Cooperative Multiagent Systemscited by 7
  26. 2024Collaborative AI Teaming in Unknown Environments via Active Goal Deductioncited by 6
  27. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolutioncited by 6
  28. 2021Dealing with Non-Stationarity in MARL via Trust-Region Decompositioncited by 5
  29. 2021MARL with General Utilities via Decentralized Shadow Reward Actor-Criticcited by 5
  30. 2021Model-based Reinforcement Learning for Service Mesh Fault Resiliency in a Web Application-levelcited by 5
  31. 2022Competition over data: how does data purchase affect users?cited by 5
  32. 2023Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibilitycited by 5
  33. 2024Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Surveycited by 5
  34. 2023Multi-Timescale Control and Communications with Deep Reinforcement Learning -- Part II: Control-Aware Radio Resource Allocationcited by 4
  35. 2021Balancing Rational and Other-Regarding Preferences in Cooperative-Competitive Environmentscited by 3
  36. 2021Learning to Robustly Negotiate Bi-Directional Lane Usage in High-Conflict Driving Scenarioscited by 3
  37. 2021Shaping Advice in Deep Multi-Agent Reinforcement Learningcited by 3
  38. 2021Learning Emergent Random Access Protocol for LEO Satellite Networkscited by 3

50 most cited of 114 All 114 (JSON)