# Revisiting the Master-Slave Architecture in Multi-Agent Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1712.07305 · HTML: https://mlsanity.com/paper/1712.07305/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2017.

## Built on by (most cited first, 12 of 12)

- [A Survey of Multi-Agent Deep Reinforcement Learning with Communication](https://mlsanity.com/paper/2203.08975/) (2022)
- [DeepAnalyze: Agentic Large Language Models for Autonomous Data Science](https://mlsanity.com/paper/2510.16872/) (2025)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Distributed Reinforcement Learning for Robot Teams: A Review](https://mlsanity.com/paper/2204.03516/) (2022)
- [Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization](https://mlsanity.com/paper/2202.01334/) (2022)
- [The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs](https://mlsanity.com/paper/2602.11583/) (2026)
- [CoDreamer: Communication-Based Decentralised World Models](https://mlsanity.com/paper/2406.13600/) (2024)
- [Hierarchically Structured Scheduling and Execution of Tasks in a Multi-Agent Environment](https://mlsanity.com/paper/2203.03021/) (2022)
- [Decentralized Coordination in Partially Observable Queueing Networks](https://mlsanity.com/paper/2208.13621/) (2022)
- [Master-slave Deep Architecture for Top-K Multi-armed Bandits with Non-linear Bandit Feedback and Diversity Constraints](https://mlsanity.com/paper/2308.12680/) (2023)
- [Finding the Weakest Link: Adversarial Attack against Multi-Agent Communications](https://mlsanity.com/paper/2605.13170/) (2026)
- [Position: Beyond Training Time, Test-Time Coordination is Essential for Cooperative MARL](https://mlsanity.com/paper/2610.17409/) (2026)

## Abstract (arXiv)

Many tasks in artificial intelligence require the collaboration of multiple agents. We exam deep reinforcement learning for multi-agent domains. Recent research efforts often take the form of two seemingly conflicting perspectives, the decentralized perspective, where each agent is supposed to have its own controller; and the centralized perspective, where one assumes there is a larger model controlling all agents. In this regard, we revisit the idea of the master-slave architecture by incorporating both perspectives within one framework. Such a hierarchical structure naturally leverages advantages from one another. The idea of combining both perspectives is intuitive and can be well motivated from many real world systems, however, out of a variety of possible realizations, we highlights three key ingredients, i.e. composed action representation, learnable communication and independent reasoning. With network designs to facilitate these explicitly, our proposal consistently outperforms latest competing methods both in synthetic experiments and when applied to challenging StarCraft micromanagement tasks.