# PooL: Pheromone-inspired Communication Framework forLarge Scale Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2202.09722 · HTML: https://mlsanity.com/paper/2202.09722/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 20 Feb 2022.

## Builds on (most cited first, 19 of 19)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [TarMAC: Targeted Multi-Agent Communication](https://mlsanity.com/paper/1810.11187/) (2018)
- [Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.05438/) (2018)
- [Learning Attentional Communication for Multi-Agent Cooperation](https://mlsanity.com/paper/1805.07733/) (2018)
- [Graph Convolutional Reinforcement Learning](https://mlsanity.com/paper/1810.09202/) (2018)
- [MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence](https://mlsanity.com/paper/1712.00600/) (2017)
- [VAIN: Attentional Multi-agent Predictive Modeling](https://mlsanity.com/paper/1706.06122/) (2017)
- [From Few to More: Large-scale Dynamic Multiagent Curriculum Learning](https://mlsanity.com/paper/1909.02790/) (2019)
- [Multiagent Soft Q-Learning](https://mlsanity.com/paper/1804.09817/) (2018)
- [Factorized Q-Learning for Large-Scale Multi-Agent Systems](https://mlsanity.com/paper/1809.03738/) (2018)
- [Stigmergic Independent Reinforcement Learning for Multi-Agent Collaboration](https://mlsanity.com/paper/1911.12504/) (2019)
- [Brain-Inspired Stigmergy Learning](https://mlsanity.com/paper/1811.08210/) (2018)

## Abstract (arXiv)

Being difficult to scale poses great problems in multi-agent coordination. Multi-agent Reinforcement Learning (MARL) algorithms applied in small-scale multi-agent systems are hard to extend to large-scale ones because the latter is far more dynamic and the number of interactions increases exponentially with the growing number of agents. Some swarm intelligence algorithms simulate the release and utilization mechanism of pheromones to control large-scale agent coordination. Inspired by such algorithms, PooL, an pheromone-based indirect communication framework applied to large scale multi-agent reinforcement learning is proposed in order to solve the large-scale multi-agent coordination problem. Pheromones released by agents of PooL are defined as outputs of most reinforcement learning algorithms, which reflect agents' views of the current environment. The pheromone update mechanism can efficiently organize the information of all agents and simplify the complex interactions among agents into low-dimensional representations. Pheromones perceived by agents can be regarded as a summary of the views of nearby agents which can better reflect the real situation of the environment. Q-Learning is taken as our base model to implement PooL and PooL is evaluated in various large-scale cooperative environments. Experiments show agents can capture effective information through PooL and achieve higher rewards than other state-of-arts methods with lower communication costs.