# A New Framework for Multi-Agent Reinforcement Learning -- Centralized Training and Exploration with Decentralized Execution via Policy Distillation

arXiv: https://arxiv.org/abs/1910.09152 · HTML: https://mlsanity.com/paper/1910.09152/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2019.

## Built on by (most cited first, 12 of 12)

- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Survey](https://mlsanity.com/paper/2110.01411/) (2021)
- [Towards Semantic Communication Protocols: A Probabilistic Logic Perspective](https://mlsanity.com/paper/2207.03920/) (2022)
- [Attention Actor-Critic algorithm for Multi-Agent Constrained Co-operative Reinforcement Learning](https://mlsanity.com/paper/2101.02349/) (2021)
- [Relative Distributed Formation and Obstacle Avoidance with Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2111.07334/) (2021)
- [Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective](https://mlsanity.com/paper/2106.05402/) (2021)
- [Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.15245/) (2022)
- [Graph Neural Networks over the Air for Decentralized Tasks in Wireless Networks](https://mlsanity.com/paper/2302.08447/) (2023)
- [Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2302.10825/) (2023)
- [Traffic Shaping and Hysteresis Mitigation Using Deep Reinforcement Learning in a Connected Driving Environment](https://mlsanity.com/paper/2302.03141/) (2023)
- [DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2312.05783/) (2023)
- [OPTIMA: Optimized Policy for Intelligent Multi-Agent Systems Enables Coordination-Aware Autonomous Vehicles](https://mlsanity.com/paper/2410.18112/) (2024)

## Abstract (arXiv)

Deep reinforcement learning (DRL) is a booming area of artificial intelligence. Many practical applications of DRL naturally involve more than one collaborative learners, making it important to study DRL in a multi-agent context. Previous research showed that effective learning in complex multi-agent systems demands for highly coordinated environment exploration among all the participating agents. Many researchers attempted to cope with this challenge through learning centralized value functions. However, the common strategy for every agent to learn their local policies directly often fail to nurture strong inter-agent collaboration and can be sample inefficient whenever agents alter their communication channels. To address these issues, we propose a new framework known as centralized training and exploration with decentralized execution via policy distillation. Guided by this framework and the maximum-entropy learning technique, we will first train agents' policies with shared global component to foster coordinated and effective learning. Locally executable policies will be derived subsequently from the trained global policies via policy distillation. Experiments show that our new framework and algorithm can achieve significantly better performance and higher sample efficiency than a cutting-edge baseline on several multi-agent DRL benchmarks.