mlsanityData to 14 Oct 2026

MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer

arXiv 2206.10607 ↗First posted 20 Jun 2022cs.LG cs.AIcited by 34

Not on the map. Why?

Abstract (arXiv)

In this paper, we consider cooperative multi-agent reinforcement learning (MARL) with sparse reward. To tackle this problem, we propose a novel method named MASER: MARL with subgoals generated from experience replay buffer. Under the widely-used assumption of centralized training with decentralized execution and consistent Q-value decomposition for MARL, MASER automatically generates proper subgoals for multiple agents from the experience replay buffer by considering both individual Q-value and total Q-value. Then, MASER designs individual intrinsic reward for each agent based on actionable representation relevant to Q-learning so that the agents reach their subgoals while maximizing the joint action value. Numerical results show that MASER significantly outperforms StarCraft II micromanagement benchmark compared to other state-of-the-art MARL algorithms.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 20 Jun 2022.

Builds on

26 of 36 references.

References · 26

  1. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2017Counterfactual Multi-Agent Policy Gradientscited by 751
  5. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  6. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  7. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
  8. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  9. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  10. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  11. 2019MAVEN: Multi-Agent Variational Explorationcited by 185
  12. 2019A Review of Cooperative Multi-Agent Deep Reinforcement Learningcited by 147
14 more
  1. 2020ROMA: Multi-Agent Reinforcement Learning with Emergent Rolescited by 137
  2. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  3. 2020RODE: Learning Roles to Decompose Multi-Agent Taskscited by 125
  4. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 116
  5. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
  6. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  7. 2019Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learningcited by 57
  8. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  9. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 54
  10. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  11. 2019Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discoverycited by 43
  12. 2018Managing engineering systems with large state and action spaces through deep reinforcement learningcited by 32
  13. 2020A Maximum Mutual Information Framework for Multi-Agent Reinforcement Learningcited by 11
  14. 2020REMAX: Relational Representation for Multi-Agent Explorationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 10; 2024: 7; 2025: 11; 2026: 4.

34 papers.

Sort:

By area:

  1. 2023A Review of Cooperation in Multi-agent Learningcited by 29
  2. 2023RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorizationcited by 20
  3. 2022Contrastive Identity-Aware Learning for Multi-Agent Value Decompositioncited by 19
  4. 2023Parameter Sharing with Network Pruning for Scalable Multi-Agent Deep Reinforcement Learningcited by 14
  5. 2023FoX: Formation-aware exploration in multi-agent reinforcement learningcited by 9
  6. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9
  7. 2023Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learningcited by 8
  8. 2023Learning Graph-Enhanced Commander-Executor for Multi-Agent Navigationcited by 6
  9. 2024Settling Decentralized Multi-Agent Coordinated Exploration by Novelty Sharingcited by 6
  10. 2025QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning?cited by 6
  11. 2023A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learningcited by 5
  12. 2024Imagine, Initialize, and Explore: An Effective Exploration Method in Multi-Agent Reinforcement Learningcited by 5
22 more
  1. 2025An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Taskscited by 5
  2. 2024Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learningcited by 4
  3. 2025Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibrationcited by 4
  4. 2023VDFD: Multi-Agent Value Decomposition Framework with Disentangled World Modelcited by 3
  5. 2025Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Explorationcited by 3
  6. 2022Entity Divider with Language Grounding in Multi-Agent Reinforcement Learningcited by 2
  7. 2024Subequivariant Reinforcement Learning in 3D Multi-Entity Physical Environmentscited by 2
  8. 2025Trajectory-Class-Aware Multi-Agent Reinforcement Learningcited by 2
  9. 2025Focusing Influence Mechanism for Multi-Agent Reinforcement Learningcited by 2
  10. 2023Hierarchical Task Network Planning for Facilitating Cooperative Multi-Agent Reinforcement Learningcited by 1
  11. 2024Learning Communication Policies for Different Follower Behaviors in a Collaborative Reference Gamecited by 1
  12. 2025B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learningcited by 1
  13. 2025Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoalscited by 1
  14. 2025Topology Enhanced MARL for Multi-Agent Cooperative Decision-Making of CAVscited by 1
  15. 2025Learning Individual Intrinsic Reward in Multi-Agent Reinforcement Learning via Incorporating Generalized Human Expertisecited by 1
  16. 2026STAIRS-Former: Spatio-Temporal Attention with Interleaved Recursive Structure Transformer for Offline Multi-task Multi-agent Reinforcement Learningcited by 1
  17. 2023SVDE: Scalable Value-Decomposition Exploration for Cooperative Multi-Agent Reinforcement Learningcited by 0
  18. 2024Enhancing Heterogeneous Multi-Agent Cooperation in Decentralized MARL via GNN-driven Intrinsic Rewardscited by 0
  19. 2025Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learningcited by 0
  20. 2026Multi-Agent Model-Based Reinforcement Learning with Joint State-Action Learned Embeddingscited by 0
  21. 2026LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learningcited by 0
  22. 2026Policy Optimality Measurement for Multi-Vehicle Decision-Making: From Extrinsic Indicators to Intrinsic Qualitycited by 0