# Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability

arXiv: https://arxiv.org/abs/2209.10003 · HTML: https://mlsanity.com/paper/2209.10003/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 95 references that are in the map.
- First posted on arXiv 20 Sep 2022.

## Builds on (most cited first, 21 of 21)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [QPLEX: Duplex Dueling Multi-Agent Q-Learning](https://mlsanity.com/paper/2008.01062/) (2020)
- [Multiagent Cooperation and Competition with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.08779/) (2015)
- [Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols](https://mlsanity.com/paper/1502.03143/) (2015)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery](https://mlsanity.com/paper/1912.03558/) (2019)
- [Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning](https://mlsanity.com/paper/1901.08492/) (2019)
- [Deep Reinforcement Learning for Event-Driven Multi-Agent Decision Processes](https://mlsanity.com/paper/1709.06656/) (2017)
- [HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism](https://mlsanity.com/paper/2110.07246/) (2021)
- [Macro-Action-Based Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2004.08646/) (2020)
- [Option-Critic in Cooperative Multi-agent Systems](https://mlsanity.com/paper/1911.12825/) (2019)
- [Towards Learning Multi-agent Negotiations via Self-Play](https://mlsanity.com/paper/2001.10208/) (2020)
- [Robust Fruit Counting: Combining Deep Learning, Tracking, and Structure from Motion](https://mlsanity.com/paper/1804.00307/) (2018)
- [Learning Multi-Robot Decentralized Macro-Action-Based Policies via a Centralized Q-Net](https://mlsanity.com/paper/1909.08776/) (2019)
- [Spatial Intention Maps for Multi-Agent Mobile Manipulation](https://mlsanity.com/paper/2103.12710/) (2021)
- [Learning for Multi-robot Cooperation in Partially Observable Stochastic Environments with Macro-actions](https://mlsanity.com/paper/1707.07399/) (2017)
- [Planning for Decentralized Control of Multiple Robots Under Uncertainty](https://mlsanity.com/paper/1402.2871/) (2014)
- [Multi-agent Hierarchical Reinforcement Learning with Dynamic Termination](https://mlsanity.com/paper/1910.09508/) (2019)
- [Weighted Double Deep Multiagent Reinforcement Learning in Stochastic Cooperative Environments](https://mlsanity.com/paper/1802.08534/) (2018)
- [Near-Optimal Adversarial Policy Switching for Decentralized Asynchronous Multi-Agent Systems](https://mlsanity.com/paper/1710.06525/) (2017)

## Built on by (most cited first, 2 of 2)

- [Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workers](https://mlsanity.com/paper/2212.11498/) (2022)
- [Meta-learning how to Share Credit among Macro-Actions](https://mlsanity.com/paper/2506.13690/) (2025)

## Abstract (arXiv)

The state-of-the-art multi-agent reinforcement learning (MARL) methods have provided promising solutions to a variety of complex problems. Yet, these methods all assume that agents perform synchronized primitive-action executions so that they are not genuinely scalable to long-horizon real-world multi-agent/robot tasks that inherently require agents/robots to asynchronously reason about high-level action selection at varying time durations. The Macro-Action Decentralized Partially Observable Markov Decision Process (MacDec-POMDP) is a general formalization for asynchronous decision-making under uncertainty in fully cooperative multi-agent tasks. In this thesis, we first propose a group of value-based RL approaches for MacDec-POMDPs, where agents are allowed to perform asynchronous learning and decision-making with macro-action-value functions in three paradigms: decentralized learning and control, centralized learning and control, and centralized training for decentralized execution (CTDE). Building on the above work, we formulate a set of macro-action-based policy gradient algorithms under the three training paradigms, where agents are allowed to directly optimize their parameterized policies in an asynchronous manner. We evaluate our methods both in simulation and on real robots over a variety of realistic domains. Empirical results demonstrate the superiority of our approaches in large multi-agent problems and validate the effectiveness of our algorithms for learning high-quality and asynchronous solutions with macro-actions.