mlsanityData to 7 Oct 2026

Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learning

arXiv 2302.09605 ↗First posted 19 Feb 2023cs.LG cs.MAcited by 2

On the map: Offline RL and imitation learning from data → Offline multi-agent RL

Abstract (arXiv)

Utilizing messages from teammates can improve coordination in cooperative Multi-agent Reinforcement Learning (MARL). Previous works typically combine raw messages of teammates with local information as inputs for policy. However, neglecting message aggregation poses significant inefficiency for policy learning. Motivated by recent advances in representation learning, we argue that efficient message aggregation is essential for good coordination in cooperative MARL. In this paper, we propose Multi-Agent communication via Self-supervised Information Aggregation (MASIA), where agents can aggregate the received messages into compact representations with high relevance to augment the local policy. Specifically, we design a permutation invariant message encoder to generate common information-aggregated representation from messages and optimize it via reconstructing and shooting future information in a self-supervised manner. Hence, each agent would utilize the most relevant parts of the aggregated representation for decision-making by a novel message extraction mechanism. Furthermore, considering the potential of offline learning for real-world applications, we build offline benchmarks for multi-agent communication, which is the first as we know. Empirical results demonstrate the superiority of our method in both online and offline settings. We also release the built offline benchmarks in this paper as a testbed for communication ability validation to facilitate further future research.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline multi-agent RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 58 of its 92 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

58 of 92 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning28 Nov
  2. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification22 Nov
  3. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning7 Jun

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  10. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  11. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  12. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
38 more
  1. 2019The StarCraft Multi-Agent Challengecited by 589
  2. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  3. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  4. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  5. 2019QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learningcited by 372
  6. 2016Learning Multiagent Communication with Backpropagationcited by 355
  7. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  8. 2020QPLEX: Duplex Dueling Multi-Agent Q-Learningcited by 267
  9. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  10. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  11. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 207
  12. 2018TarMAC: Targeted Multi-Agent Communicationcited by 201
  13. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  14. 2018A Survey and Critique of Multiagent Deep Reinforcement Learningcited by 173
  15. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 168
  16. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 153
  17. 2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 147
  18. 2019An Attentive Survey of Attention Modelscited by 123
  19. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  20. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  21. 2022A Survey of Multi-Agent Deep Reinforcement Learning with Communicationcited by 103
  22. 2020Off-Policy Multi-Agent Decomposed Policy Gradientscited by 97
  23. 2019Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learningcited by 90
  24. 2019Learning Nearly Decomposable Value Functions Via Communication Minimizationcited by 85
  25. 2021Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharingcited by 85
  26. 2019Shapley Q-value: A Local Reward Approach to Solve Global Reward Gamescited by 71
  27. 2020Learning Individually Inferred Communication for Multi-Agent Cooperationcited by 55
  28. 2021Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learningcited by 54
  29. 2021Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networkscited by 52
  30. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  31. 2019Biases for Emergent Communication in Multi-agent Reinforcement Learningcited by 48
  32. 2019Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Controlcited by 46
  33. 2021Learning to Ground Multi-Agent Communication with Autoencoderscited by 37
  34. 2022Multi-agent Dynamic Algorithm Configurationcited by 36
  35. 2021Regularized Behavior Value Estimationcited by 32
  36. 2021Offline Decentralized Multi-Agent Reinforcement Learningcited by 32
  37. 2019Learning Agent Communication under Limited Bandwidth by Message Pruningcited by 31
  38. 2020Succinct and Robust Multi-Agent Communication With Temporal Message Controlcited by 29

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Robust and Efficient Communication in Multi-Agent Reinforcement Learningcited by 2
  2. 2026HiComm: Hierarchical Communication for Multi-agent Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs
  2. 2026LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learning
  3. 2025Communicating Plans, Not Percepts: Scalable Multi-Agent Coordination with Embodied World Models
  4. 2026Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning
  5. 2024Verco: Learning Coordinated Verbal Communication for Multi-agent Reinforcement Learning
  6. 2026Multi-Agent Decision-Focused Learning via Value-Aware Sequential Communication
  7. 2021Learning Selective Communication for Multi-Agent Path Finding
  8. 2026Active Learning for Communication Structure Optimization in LLM-Based Multi-Agent Systems