# Learning Multi-Robot Decentralized Macro-Action-Based Policies via a Centralized Q-Net

arXiv: https://arxiv.org/abs/1909.08776 · HTML: https://mlsanity.com/paper/1909.08776/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2019.

## Built on by (most cited first, 12 of 12)

- [Partially Observable Markov Decision Processes in Robotics: A Survey](https://mlsanity.com/paper/2209.10342/) (2022)
- [Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.04402/) (2021)
- [Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense](https://mlsanity.com/paper/2410.17351/) (2024)
- [Distributed Reinforcement Learning for Robot Teams: A Review](https://mlsanity.com/paper/2204.03516/) (2022)
- [Spatial Intention Maps for Multi-Agent Mobile Manipulation](https://mlsanity.com/paper/2103.12710/) (2021)
- [On Centralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2408.14597/) (2024)
- [An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks](https://mlsanity.com/paper/2502.04773/) (2025)
- [Local Advantage Actor-Critic for Robust Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2110.08642/) (2021)
- [Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2209.10003/) (2022)
- [Maximum Entropy Dueling Network Architecture in Atari Domain](https://mlsanity.com/paper/2107.14457/) (2021)
- [Formal Modelling for Multi-Robot Systems Under Uncertainty](https://mlsanity.com/paper/2305.17018/) (2023)
- [Cross-Modal Navigation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.06595/) (2026)

## Abstract (arXiv)

In many real-world multi-robot tasks, high-quality solutions often require a team of robots to perform asynchronous actions under decentralized control. Decentralized multi-agent reinforcement learning methods have difficulty learning decentralized policies because of the environment appearing to be non-stationary due to other agents also learning at the same time. In this paper, we address this challenge by proposing a macro-action-based decentralized multi-agent double deep recurrent Q-net (MacDec-MADDRQN) which trains each decentralized Q-net using a centralized Q-net for action selection. A generalized version of MacDec-MADDRQN with two separate training environments, called Parallel-MacDec-MADDRQN, is also presented to leverage either centralized or decentralized exploration. The advantages and the practical nature of our methods are demonstrated by achieving near-centralized results in simulation and having real robots accomplish a warehouse tool delivery task in an efficient way.