mlsanityData to 14 Oct 2026

Higher Replay Ratio Empowers Sample-Efficient Multi-Agent Reinforcement Learning

arXiv 2404.09715 ↗First posted 15 Apr 2024cs.LG cs.AI cs.MAcited by 2

Not on the map. Why?

Abstract (arXiv)

One of the notorious issues for Reinforcement Learning (RL) is poor sample efficiency. Compared to single agent RL, the sample efficiency for Multi-Agent Reinforcement Learning (MARL) is more challenging because of its inherent partial observability, non-stationary training, and enormous strategy space. Although much effort has been devoted to developing new methods and enhancing sample efficiency, we look at the widely used episodic training mechanism. In each training step, tens of frames are collected, but only one gradient step is made. We argue that this episodic training could be a source of poor sample efficiency. To better exploit the data already collected, we propose to increase the frequency of the gradient updates per environment interaction (a.k.a. Replay Ratio or Update-To-Data ratio). To show its generality, we evaluate 3 MARL methods on 6 SMAC tasks. The empirical results validate that a higher replay ratio significantly improves the sample efficiency for MARL algorithms. The codes to reimplement the results presented in this paper are open-sourced at https://anonymous.4open.science/r/rrforMARL-0D83/.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

22 of 31 references.

References · 22

  1. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  2. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  3. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  5. 2019The StarCraft Multi-Agent Challengecited by 597
  6. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  7. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  8. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  9. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  10. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  11. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  12. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
10 more
  1. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  2. 2021Celebrating Diversity in Shared Multi-Agent Reinforcement Learningcited by 98
  3. 2023MarioGPT: Open-Ended Text2Level Generation through Large Language Modelscited by 64
  4. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  5. 2023Small batch deep reinforcement learningcited by 36
  6. 2023Dynamic Mesh-Aware Radiance Fieldscited by 21
  7. 2023NA2Q: Neural Attention Additive Model for Interpretable Multi-Agent Q-Learningcited by 14
  8. 2023The Five-Dollar Model: Generating Game Maps and Sprites from Sentence Embeddingscited by 14
  9. 2022MIXRTs: Toward Interpretable Multi-Agent Reinforcement Learning via Mixing Recurrent Soft Decision Treescited by 11
  10. 2019Non-Smooth Newton Methods for Deformable Multi-Body Dynamicscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Sample-efficient LLM Optimization with Reset Replaycited by 4
  2. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026When Do Multi-Agent Systems Outperform? Analysing the Learning Efficiency of Agentic Systems
  2. 2026When Does Non-Uniform Replay Matter in Reinforcement Learning?
  3. 2024Efficient Diversity-based Experience Replay for Deep Reinforcement Learning
  4. 2025Centralized Adaptive Sampling for Reliable Co-Training of Independent Multi-Agent Policies
  5. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  6. 2025Ensemble-MIX: Enhancing Sample-Efficiency in Multi-Agent RL via Selective Ensemble-Based Exploration
  7. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  8. 2024EfficientZero V2: Mastering Discrete and Continuous Control with Limited Data