# Beyond Joint Demonstrations: Personalized Expert Guidance for Efficient Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2403.08936 · HTML: https://mlsanity.com/paper/2403.08936/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 15 of 15)

- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration](https://mlsanity.com/paper/2202.04628/) (2022)
- [Heterogeneous Multi-Robot Reinforcement Learning](https://mlsanity.com/paper/2301.07137/) (2023)
- [Learning Altruistic Behaviours in Reinforcement Learning without External Rewards](https://mlsanity.com/paper/2107.09598/) (2021)
- [Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.17540/) (2022)
- [DM2: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching](https://mlsanity.com/paper/2206.00233/) (2022)
- [Policy Gradient from Demonstration and Curiosity](https://mlsanity.com/paper/2004.10430/) (2020)
- [Sample-Efficient Multi-Agent Reinforcement Learning with Demonstrations for Flocking Control](https://mlsanity.com/paper/2209.08351/) (2022)

## Built on by (most cited first, 5 of 5)

- [Sketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketches](https://mlsanity.com/paper/2503.11918/) (2025)
- [LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems](https://mlsanity.com/paper/2511.19368/) (2025)
- [PLANRL: A Motion Planning and Imitation Learning Framework to Bootstrap Reinforcement Learning](https://mlsanity.com/paper/2408.04054/) (2024)
- [Adaptive Conformal Guidance for Learning under Uncertainty](https://mlsanity.com/paper/2502.16736/) (2025)
- [From Solo to Symphony: Orchestrating Multi-Agent Collaboration with Single-Agent Demos](https://mlsanity.com/paper/2511.02762/) (2025)

## Abstract (arXiv)

Multi-Agent Reinforcement Learning (MARL) algorithms face the challenge of efficient exploration due to the exponential increase in the size of the joint state-action space. While demonstration-guided learning has proven beneficial in single-agent settings, its direct applicability to MARL is hindered by the practical difficulty of obtaining joint expert demonstrations. In this work, we introduce a novel concept of personalized expert demonstrations, tailored for each individual agent or, more broadly, each individual type of agent within a heterogeneous team. These demonstrations solely pertain to single-agent behaviors and how each agent can achieve personal goals without encompassing any cooperative elements, thus naively imitating them will not achieve cooperation due to potential conflicts. To this end, we propose an approach that selectively utilizes personalized expert demonstrations as guidance and allows agents to learn to cooperate, namely personalized expert-guided MARL (PegMARL). This algorithm utilizes two discriminators: the first provides incentives based on the alignment of individual agent behavior with demonstrations, and the second regulates incentives based on whether the behaviors lead to the desired outcome. We evaluate PegMARL using personalized demonstrations in both discrete and continuous environments. The experimental results demonstrate that PegMARL outperforms state-of-the-art MARL algorithms in solving coordinated tasks, achieving strong performance even when provided with suboptimal personalized demonstrations. We also showcase PegMARL's capability of leveraging joint demonstrations in the StarCraft scenario and converging effectively even with demonstrations from non-co-trained policies.