# Decentralized Multi-Agent Reinforcement Learning with Global State Prediction

arXiv: https://arxiv.org/abs/2306.12926 · HTML: https://mlsanity.com/paper/2306.12926/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 22 Jun 2023.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [A Survey and Critique of Multiagent Deep Reinforcement Learning](https://mlsanity.com/paper/1810.05587/) (2018)
- [Multi-task Deep Reinforcement Learning with PopArt](https://mlsanity.com/paper/1809.04474/) (2018)
- [Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1906.04737/) (2019)
- [Loss is its own Reward: Self-Supervision for Reinforcement Learning](https://mlsanity.com/paper/1612.07307/) (2016)
- [Modeling Others using Oneself in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1802.09640/) (2018)
- [Reducing Overestimation Bias in Multi-Agent Domains Using Double Centralized Critics](https://mlsanity.com/paper/1910.01465/) (2019)
- [Emergent Social Learning via Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2010.00581/) (2020)
- [Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregation](https://mlsanity.com/paper/2211.02127/) (2022)
- [Agent Modeling as Auxiliary Task for Deep Reinforcement Learning](https://mlsanity.com/paper/1907.09597/) (2019)
- [Spatial Intention Maps for Multi-Agent Mobile Manipulation](https://mlsanity.com/paper/2103.12710/) (2021)

## Built on by (most cited first, 3 of 3)

- [Multi-Agent Reinforcement Learning-Based UAV Pathfinding for Obstacle Avoidance in Stochastic Environment](https://mlsanity.com/paper/2310.16659/) (2023)
- [Structured Diversity Control: A Dual-Level Framework for Group-Aware Multi-Agent Coordination](https://mlsanity.com/paper/2506.18651/) (2025)
- [Value-Aware Prediction for Robust Multi-Agent Coordination Under Communication Loss](https://mlsanity.com/paper/2607.17914/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has seen remarkable success in the control of single robots. However, applying DRL to robot swarms presents significant challenges. A critical challenge is non-stationarity, which occurs when two or more robots update individual or shared policies concurrently, thereby engaging in an interdependent training process with no guarantees of convergence. Circumventing non-stationarity typically involves training the robots with global information about other agents' states and/or actions. In contrast, in this paper we explore how to remove the need for global information. We pose our problem as a Partially Observable Markov Decision Process, due to the absence of global knowledge on other agents. Using collective transport as a testbed scenario, we study two approaches to multi-agent training. In the first, the robots exchange no messages, and are trained to rely on implicit communication through push-and-pull on the object to transport. In the second approach, we introduce Global State Prediction (GSP), a network trained to forma a belief over the swarm as a whole and predict its future states. We provide a comprehensive study over four well-known deep reinforcement learning algorithms in environments with obstacles, measuring performance as the successful transport of the object to the goal within a desired time-frame. Through an ablation study, we show that including GSP boosts performance and increases robustness when compared with methods that use global knowledge.