# Learning for Multi-robot Cooperation in Partially Observable Stochastic Environments with Macro-actions

arXiv: https://arxiv.org/abs/1707.07399 · HTML: https://mlsanity.com/paper/1707.07399/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2017.

## Built on by (most cited first, 8 of 8)

- [Sample-Efficient Reinforcement Learning of Partially Observable Markov Games](https://mlsanity.com/paper/2206.01315/) (2022)
- [Centralized Model and Exploration Policy for Multi-Agent RL](https://mlsanity.com/paper/2107.06434/) (2021)
- [Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2209.10003/) (2022)
- [Socratic: Enhancing Human Teamwork via AI-enabled Coaching](https://mlsanity.com/paper/2502.17643/) (2025)
- [A Multi-UAV System for Exploration and Target Finding in Cluttered and GPS-Denied Environments](https://mlsanity.com/paper/2107.08834/) (2021)
- [HDDLGym: A Tool for Studying Multi-Agent Hierarchical Problems Defined in HDDL with OpenAI Gym](https://mlsanity.com/paper/2505.22597/) (2025)
- [Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with k-step Policy Gradients](https://mlsanity.com/paper/2605.10909/) (2026)
- [Formal Modelling for Multi-Robot Systems Under Uncertainty](https://mlsanity.com/paper/2305.17018/) (2023)

## Abstract (arXiv)

This paper presents a data-driven approach for multi-robot coordination in partially-observable domains based on Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) and macro-actions (MAs). Dec-POMDPs provide a general framework for cooperative sequential decision making under uncertainty and MAs allow temporally extended and asynchronous action execution. To date, most methods assume the underlying Dec-POMDP model is known a priori or a full simulator is available during planning time. Previous methods which aim to address these issues suffer from local optimality and sensitivity to initial conditions. Additionally, few hardware demonstrations involving a large team of heterogeneous robots and with long planning horizons exist. This work addresses these gaps by proposing an iterative sampling based Expectation-Maximization algorithm (iSEM) to learn polices using only trajectory data containing observations, MAs, and rewards. Our experiments show the algorithm is able to achieve better solution quality than the state-of-the-art learning-based methods. We implement two variants of multi-robot Search and Rescue (SAR) domains (with and without obstacles) on hardware to demonstrate the learned policies can effectively control a team of distributed robots to cooperate in a partially observable stochastic environment.