mlsanityData to 14 Oct 2026

Multi-agent Continual Coordination via Progressive Task Contextualization

arXiv 2305.13937 ↗First posted 7 May 2023cs.MA cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Cooperative Multi-agent Reinforcement Learning (MARL) has attracted significant attention and played the potential for many real-world applications. Previous arts mainly focus on facilitating the coordination ability from different aspects (e.g., non-stationarity, credit assignment) in single-task or multi-task scenarios, ignoring the stream of tasks that appear in a continual manner. This ignorance makes the continual coordination an unexplored territory, neither in problem formulation nor efficient algorithms designed. Towards tackling the mentioned issue, this paper proposes an approach Multi-Agent Continual Coordination via Progressive Task Contextualization, dubbed MACPro. The key point lies in obtaining a factorized policy, using shared feature extraction layers but separated independent task heads, each specializing in a specific class of tasks. The task heads can be progressively expanded based on the learned task contextualization. Moreover, to cater to the popular CTDE paradigm in MARL, each agent learns to predict and adopt the most relevant policy head based on local information in a decentralized manner. We show in multiple multi-agent benchmarks that existing continual learning methods fail, while MACPro is able to achieve close-to-optimal performance. More results also disclose the effectiveness of MACPro from multiple aspects like high generalization ability.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 52 of its 81 references that are in the map. First posted on arXiv 7 May 2023.

Builds on

52 of 81 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  7. 2018Experience Replay for Continual Learningcited by 1,095
  8. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  9. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 708
  10. 2019The StarCraft Multi-Agent Challengecited by 597
  11. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  12. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 449
38 more
  1. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 379
  2. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  3. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 270
  4. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  5. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  6. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 155
  7. 2019A Review of Cooperative Multi-Agent Deep Reinforcement Learningcited by 147
  8. 2020A Neural Dirichlet Process Mixture Model for Task-Free Continual Learningcited by 134
  9. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
  10. 2022A Survey of Multi-Agent Deep Reinforcement Learning with Communicationcited by 104
  11. 2019Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learningcited by 93
  12. 2018PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learningcited by 90
  13. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 87
  14. 2022A Survey on Model-based Reinforcement Learningcited by 66
  15. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 56
  16. 2021Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networkscited by 54
  17. 2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 49
  18. 2020Continual Model-Based Reinforcement Learning with Hypernetworkscited by 44
  19. 2021Same State, Different Task: Continual Reinforcement Learning without Interferencecited by 43
  20. 2019Policy Consolidation for Continual Reinforcement Learningcited by 39
  21. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39
  22. 2020Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learningcited by 38
  23. 2021CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agentscited by 38
  24. 2022Multi-agent Dynamic Algorithm Configurationcited by 36
  25. 2021Continuous Coordination As a Realistic Scenario for Lifelong Learningcited by 32
  26. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28
  27. 2020Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorizationcited by 26
  28. 2021Block Contextual MDPs for Continual Learningcited by 25
  29. 2020Lipschitz Lifelong Reinforcement Learningcited by 23
  30. 2020Lifelong Incremental Reinforcement Learning with Online Bayesian Inferencecited by 23
  31. 2022Towards Comprehensive Testing on the Robustness of Cooperative Multi-agent Reinforcement Learningcited by 21
  32. 2022Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospectscited by 17
  33. 2022The Effectiveness of World Models for Continual Reinforcement Learningcited by 17
  34. 2019SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multi-Agent Reinforcement Learningcited by 15
  35. 2022Multi-Agent Policy Transfer via Task Relationship Modelingcited by 14
  36. 2022Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualizationcited by 14
  37. 2022Heterogeneous Multi-agent Zero-Shot Coordination by Coevolutioncited by 12
  38. 2021LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammatescited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 6.

11 papers.

  1. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  2. 2025A Survey of Continual Reinforcement Learningcited by 18
  3. 2025MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learningcited by 11
  4. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 1
  5. 2026Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networkscited by 1
  6. 2024FlickerFusion: Intra-trajectory Domain Generalizing Multi-Agent RLcited by 0
  7. 2026Continual Quadruped Robots Coordination via Semantic Skill Discoverycited by 0
  8. 2026Offline Multi-agent Continual Cooperation via Skill Partition and Reusecited by 0
  9. 2026PRIME: Plasticity Recovery in Multi-Agent Environments for UAV-Assisted Emergency Communication Networkscited by 0
  10. 2026A MARL Centered Reference Architecture for Large Language Model Augmentation in Smart Manufacturingcited by 0
  11. 2026Continual Graph Multi-Agent Reinforcement Learningcited by 0