# CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning

arXiv: https://arxiv.org/abs/1809.05188 · HTML: https://mlsanity.com/paper/1809.05188/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2018.

## Built on by (most cited first, 30 of 31)

- [Safe Multi-Agent Reinforcement Learning via Shielding](https://mlsanity.com/paper/2101.11196/) (2021)
- [Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.04402/) (2021)
- [Position: Theory of Mind Benchmarks are Broken for Large Language Models](https://mlsanity.com/paper/2412.19726/) (2024)
- [Unbiased Asymmetric Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2105.11674/) (2021)
- [Learning Efficient Multi-Agent Cooperative Visual Exploration](https://mlsanity.com/paper/2110.05734/) (2021)
- [ALMA: Hierarchical Learning for Composite Multi-Agent Tasks](https://mlsanity.com/paper/2205.14205/) (2022)
- [Asynchronous Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2209.10113/) (2022)
- [Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration](https://mlsanity.com/paper/2301.03398/) (2023)
- [Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming](https://mlsanity.com/paper/2201.08484/) (2022)
- [Dealing With Non-stationarity in Decentralized Cooperative Multi-Agent Deep Reinforcement Learning via Multi-Timescale Learning](https://mlsanity.com/paper/2302.02792/) (2023)
- [Collaborative Target Search with a Visual Drone Swarm: An Adaptive Curriculum Embedded Multistage Reinforcement Learning Approach](https://mlsanity.com/paper/2204.12181/) (2022)
- [Multi-Agent Curricula and Emergent Implicit Signaling](https://mlsanity.com/paper/2106.11156/) (2021)
- [Mediated Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.08419/) (2023)
- [Deception in Social Learning: A Multi-Agent Reinforcement Learning Perspective](https://mlsanity.com/paper/2106.05402/) (2021)
- [Negotiative Alignment: Embracing Disagreement to Achieve Fairer Outcomes -- Insights from Urban Studies](https://mlsanity.com/paper/2503.12613/) (2025)
- [Learning to Robustly Negotiate Bi-Directional Lane Usage in High-Conflict Driving Scenarios](https://mlsanity.com/paper/2103.12070/) (2021)
- [Local Advantage Actor-Critic for Robust Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2110.08642/) (2021)
- [Residual Q-Networks for Value Function Factorizing in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.15245/) (2022)
- [ChemGymRL: An Interactive Framework for Reinforcement Learning for Digital Chemistry](https://mlsanity.com/paper/2305.14177/) (2023)
- [Recursive Reasoning Graph for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.02844/) (2022)
- [SwiftRL: Towards Efficient Reinforcement Learning on Real Processing-In-Memory Systems](https://mlsanity.com/paper/2405.03967/) (2024)
- [Learning Multiple Coordinated Agents under Directed Acyclic Graph Constraints](https://mlsanity.com/paper/2307.07529/) (2023)
- [DPMAC: Differentially Private Communication for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2308.09902/) (2023)
- [Responsible Emergent Multi-Agent Behavior](https://mlsanity.com/paper/2311.01609/) (2023)
- [Curriculum Imitation Learning of Distributed Multi-Robot Policies](https://mlsanity.com/paper/2509.25097/) (2025)
- [Solving Collaborative Dec-POMDPs with Deep Reinforcement Learning Heuristics](https://mlsanity.com/paper/2211.15411/) (2022)
- [Actor-Director-Critic: A Novel Deep Reinforcement Learning Framework](https://mlsanity.com/paper/2301.03887/) (2023)
- [Scalable Multi-agent Reinforcement Learning for Factory-wide Dynamic Scheduling](https://mlsanity.com/paper/2409.13571/) (2024)
- [Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2505.08630/) (2025)
- [COIN: Collaborative Interaction-Aware Multi-Agent Reinforcement Learning for Self-Driving Systems](https://mlsanity.com/paper/2603.24931/) (2026)

## Abstract (arXiv)

A variety of cooperative multi-agent control problems require agents to achieve individual goals while contributing to collective success. This multi-goal multi-agent setting poses difficulties for recent algorithms, which primarily target settings with a single global reward, due to two new challenges: efficient exploration for learning both individual goal attainment and cooperation for others' success, and credit-assignment for interactions between actions and goals of different agents. To address both challenges, we restructure the problem into a novel two-stage curriculum, in which single-agent goal attainment is learned prior to learning multi-agent cooperation, and we derive a new multi-goal multi-agent policy gradient with a credit function for localized credit assignment. We use a function augmentation scheme to bridge value and policy functions across the curriculum. The complete architecture, called CM3, learns significantly faster than direct adaptations of existing algorithms on three challenging multi-goal multi-agent problems: cooperative navigation in difficult formations, negotiating multi-vehicle lane changes in the SUMO traffic simulator, and strategic cooperation in a Checkers environment.