# Option-Critic in Cooperative Multi-agent Systems

arXiv: https://arxiv.org/abs/1911.12825 · HTML: https://mlsanity.com/paper/1911.12825/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2019.

## Built on by (most cited first, 17 of 17)

- [Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.04402/) (2021)
- [Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.10113/) (2022)
- [On Centralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2408.14597/) (2024)
- [Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs](https://mlsanity.com/paper/2307.11629/) (2023)
- [Optimizing delegation between human and AI collaborative agents](https://mlsanity.com/paper/2309.14718/) (2023)
- [Variational Offline Multi-agent Skill Discovery](https://mlsanity.com/paper/2405.16386/) (2024)
- [Learning Multi-agent Skills for Tabular Reinforcement Learning using Factor Graphs](https://mlsanity.com/paper/2201.08227/) (2022)
- [Optimizing Delegation in Collaborative Human-AI Hybrid Teams](https://mlsanity.com/paper/2402.05605/) (2024)
- [Multi-Agent Reinforcement Learning with a Hierarchy of Reward Machines](https://mlsanity.com/paper/2403.07005/) (2024)
- [Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2209.10003/) (2022)
- [Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperation](https://mlsanity.com/paper/2206.12718/) (2022)
- [Optimizing Risk-averse Human-AI Hybrid Teams](https://mlsanity.com/paper/2403.08386/) (2024)
- [Comprehend, Divide, and Conquer: Feature Subspace Exploration via Multi-Agent Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2504.17356/) (2025)
- [Coordinating Policies Among Multiple Agents via an Intelligent Communication Channel](https://mlsanity.com/paper/2205.10607/) (2022)
- [Multi-agent Deep Covering Skill Discovery](https://mlsanity.com/paper/2210.03269/) (2022)
- [Hierarchical Reinforcement Learning for Optimal Agent Grouping in Cooperative Systems](https://mlsanity.com/paper/2501.06554/) (2025)
- [Enhancing Multi-Agent Collaboration with Attention-Based Actor-Critic Policies](https://mlsanity.com/paper/2507.22782/) (2025)

## Abstract (arXiv)

In this paper, we investigate learning temporal abstractions in cooperative multi-agent systems, using the options framework (Sutton et al, 1999). First, we address the planning problem for the decentralized POMDP represented by the multi-agent system, by introducing a common information approach. We use the notion of common beliefs and broadcasting to solve an equivalent centralized POMDP problem. Then, we propose the Distributed Option Critic (DOC) algorithm, which uses centralized option evaluation and decentralized intra-option improvement. We theoretically analyze the asymptotic convergence of DOC and build a new multi-agent environment to demonstrate its validity. Our experiments empirically show that DOC performs competitively against baselines and scales with the number of agents.