# Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2210.06274 · HTML: https://mlsanity.com/paper/2210.06274/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 29 of 29)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks](https://mlsanity.com/paper/2006.07869/) (2020)
- [Learning Attentional Communication for Multi-Agent Cooperation](https://mlsanity.com/paper/1805.07733/) (2018)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks](https://mlsanity.com/paper/1812.09755/) (2018)
- [A Survey of Multi-Agent Deep Reinforcement Learning with Communication](https://mlsanity.com/paper/2203.08975/) (2022)
- [Opponent Modeling in Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05559/) (2016)
- [Learning to Schedule Communication in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/1902.01554/) (2019)
- [Learning Latent Representations to Influence Multi-Agent Interaction](https://mlsanity.com/paper/2011.06619/) (2020)
- [A Survey on Traffic Signal Control Methods](https://mlsanity.com/paper/1904.08117/) (2019)
- [Learning Efficient Multi-agent Communication: An Information Bottleneck Approach](https://mlsanity.com/paper/1911.06992/) (2019)
- [Agent Modelling under Partial Observability for Deep Reinforcement Learning](https://mlsanity.com/paper/2006.09447/) (2020)
- [Efficient Communication in Multi-Agent Reinforcement Learning via Variance Based Control](https://mlsanity.com/paper/1909.02682/) (2019)
- [R-MADDPG for Partially Observable Environments and Limited Communication](https://mlsanity.com/paper/2002.06684/) (2020)
- [Succinct and Robust Multi-Agent Communication With Temporal Message Control](https://mlsanity.com/paper/2010.14391/) (2020)
- [Multi-Modal Trajectory Prediction of NBA Players](https://mlsanity.com/paper/2008.07870/) (2020)
- [Message-Dropout: An Efficient Training Method for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1902.06527/) (2019)
- [Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects](https://mlsanity.com/paper/2203.10603/) (2022)
- [MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models](https://mlsanity.com/paper/2103.03662/) (2021)
- [Information State Embedding in Partially Observable Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2004.01098/) (2020)
- [Event-Triggered Multi-agent Reinforcement Learning with Communication under Limited-bandwidth Constraint](https://mlsanity.com/paper/2010.04978/) (2020)
- [Time-series Imputation of Temporally-occluded Multiagent Trajectories](https://mlsanity.com/paper/2106.04219/) (2021)

## Built on by (most cited first, 3 of 3)

- [The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs](https://mlsanity.com/paper/2602.11583/) (2026)
- [RecBayes: Recurrent Bayesian Ad Hoc Teamwork in Large Partially Observable Domains](https://mlsanity.com/paper/2506.15756/) (2025)
- [Degrees of Freedom in Multi-Agent Coordination: Between Centralized and Decentralized Regimes](https://mlsanity.com/paper/2610.17751/) (2026)

## Abstract (arXiv)

We introduce hybrid execution in multi-agent reinforcement learning (MARL), a new paradigm in which agents aim to successfully complete cooperative tasks with arbitrary communication levels at execution time by taking advantage of information-sharing among the agents. Under hybrid execution, the communication level can range from a setting in which no communication is allowed between agents (fully decentralized), to a setting featuring full communication (fully centralized), but the agents do not know beforehand which communication level they will encounter at execution time. To formalize our setting, we define a new class of multi-agent partially observable Markov decision processes (POMDPs) that we name hybrid-POMDPs, which explicitly model a communication process between the agents. We contribute MARO, an approach that makes use of an auto-regressive predictive model, trained in a centralized manner, to estimate missing agents' observations at execution time. We evaluate MARO on standard scenarios and extensions of previous benchmarks tailored to emphasize the negative impact of partial observability in MARL. Experimental results show that our method consistently outperforms relevant baselines, allowing agents to act with faulty communication while successfully exploiting shared information.