# M3RL: Mind-aware Multi-agent Management Reinforcement Learning

arXiv: https://arxiv.org/abs/1810.00147 · HTML: https://mlsanity.com/paper/1810.00147/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2018.

## Built on by (most cited first, 22 of 22)

- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [A Survey of Ad Hoc Teamwork Research](https://mlsanity.com/paper/2202.10450/) (2022)
- [AGENT: A Benchmark for Core Psychological Reasoning](https://mlsanity.com/paper/2102.12321/) (2021)
- [Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition](https://mlsanity.com/paper/2105.08692/) (2021)
- [ALMA: Hierarchical Learning for Composite Multi-Agent Tasks](https://mlsanity.com/paper/2205.14205/) (2022)
- [Oracles & Followers: Stackelberg Equilibria in Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.11942/) (2022)
- [Learning to Mitigate AI Collusion on Economic Platforms](https://mlsanity.com/paper/2202.07106/) (2022)
- [Information Design in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.06807/) (2023)
- [Multi-Sender Persuasion: A Computational Perspective](https://mlsanity.com/paper/2402.04971/) (2024)
- [Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning](https://mlsanity.com/paper/2406.08002/) (2024)
- [Modelling Cooperation in Network Games with Spatio-Temporal Complexity](https://mlsanity.com/paper/2102.06911/) (2021)
- [Social World Model-Augmented Mechanism Design Policy Learning](https://mlsanity.com/paper/2510.19270/) (2025)
- [Solving Dynamic Principal-Agent Problems with a Rationally Inattentive Principal](https://mlsanity.com/paper/2202.01691/) (2022)
- [Social Environment Design](https://mlsanity.com/paper/2402.14090/) (2024)
- [Carbon Market Simulation with Adaptive Mechanism Design](https://mlsanity.com/paper/2406.07875/) (2024)
- [Structured Diversification Emergence via Reinforced Organization Control and Hierarchical Consensus Learning](https://mlsanity.com/paper/2102.04775/) (2021)
- [EMOTE: An Explainable architecture for Modelling the Other Through Empathy](https://mlsanity.com/paper/2306.00295/) (2023)
- [Learning Multiple Coordinated Agents under Directed Acyclic Graph Constraints](https://mlsanity.com/paper/2307.07529/) (2023)
- [Goals are Enough: Inducing AdHoc cooperation among unseen Multi-Agent systems in IMFs](https://mlsanity.com/paper/2310.17416/) (2023)
- [Towards Adaptive IMFs -- Generalization of utility functions in Multi-Agent Frameworks](https://mlsanity.com/paper/2405.07621/) (2024)
- [ReLExS: Reinforcement Learning Explanations for Stackelberg No-Regret Learners](https://mlsanity.com/paper/2408.14086/) (2024)
- [ReCollab: Retrieval-Augmented LLMs for Cooperative Ad-hoc Teammate Modeling](https://mlsanity.com/paper/2512.22129/) (2025)

## Abstract (arXiv)

Most of the prior work on multi-agent reinforcement learning (MARL) achieves optimal collaboration by directly controlling the agents to maximize a common reward. In this paper, we aim to address this from a different angle. In particular, we consider scenarios where there are self-interested agents (i.e., worker agents) which have their own minds (preferences, intentions, skills, etc.) and can not be dictated to perform tasks they do not wish to do. For achieving optimal coordination among these agents, we train a super agent (i.e., the manager) to manage them by first inferring their minds based on both current and past observations and then initiating contracts to assign suitable tasks to workers and promise to reward them with corresponding bonuses so that they will agree to work together. The objective of the manager is maximizing the overall productivity as well as minimizing payments made to the workers for ad-hoc worker teaming. To train the manager, we propose Mind-aware Multi-agent Management Reinforcement Learning (M^3RL), which consists of agent modeling and policy learning. We have evaluated our approach in two environments, Resource Collection and Crafting, to simulate multi-agent management problems with various task settings and multiple designs for the worker agents. The experimental results have validated the effectiveness of our approach in modeling worker agents' minds online, and in achieving optimal ad-hoc teaming with good generalization and fast adaptation.