# Learning to Teach in Cooperative Multiagent Reinforcement Learning

arXiv: https://arxiv.org/abs/1805.07830 · HTML: https://mlsanity.com/paper/1805.07830/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 May 2018.

## Built on by (most cited first, 30 of 45)

- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.04402/) (2021)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Position: Theory of Mind Benchmarks are Broken for Large Language Models](https://mlsanity.com/paper/2412.19726/) (2024)
- [SHAQ: Incorporating Shapley Value Theory into Multi-Agent Q-Learning](https://mlsanity.com/paper/2105.15013/) (2021)
- [Influencing Long-Term Behavior in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2203.03535/) (2022)
- [Understanding the Complexity Gains of Single-Task RL with a Curriculum](https://mlsanity.com/paper/2212.12809/) (2022)
- [Causal Multi-Agent Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2111.06721/) (2021)
- [Multi-Agent Policy Transfer via Task Relationship Modeling](https://mlsanity.com/paper/2203.04482/) (2022)
- [HAMMER: Multi-Level Coordination of Reinforcement Learning Agents via Learned Messaging](https://mlsanity.com/paper/2102.00824/) (2021)
- [Towards More Human-like AI Communication: A Review of Emergent Communication Research](https://mlsanity.com/paper/2308.02541/) (2023)
- [A Framework for Learning to Request Rich and Contextually Useful Information from Humans](https://mlsanity.com/paper/2110.08258/) (2021)
- [Admissible Policy Teaching through Reward Design](https://mlsanity.com/paper/2201.02185/) (2022)
- [Action Advising with Advice Imitation in Deep Reinforcement Learning](https://mlsanity.com/paper/2104.08441/) (2021)
- [Explainable Action Advising for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2211.07882/) (2022)
- [On Centralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2408.14597/) (2024)
- [Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference](https://mlsanity.com/paper/2412.14355/) (2024)
- [Modelling Cooperation in Network Games with Spatio-Temporal Complexity](https://mlsanity.com/paper/2102.06911/) (2021)
- [Game-Theoretical Perspectives on Active Equilibria: A Preferred Solution Concept over Nash Equilibria](https://mlsanity.com/paper/2210.16175/) (2022)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)
- [Improving Reinforcement Learning with Human Assistance: An Argument for Human Subject Studies with HIPPO Gym](https://mlsanity.com/paper/2102.02639/) (2021)
- [MultiPrompter: Cooperative Prompt Optimization with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.16730/) (2023)
- [Few-Shot Teamwork](https://mlsanity.com/paper/2207.09300/) (2022)
- [Soft Action Priors: Towards Robust Policy Transfer](https://mlsanity.com/paper/2209.09882/) (2022)
- [Introspective Action Advising for Interpretable Transfer Learning](https://mlsanity.com/paper/2306.12314/) (2023)
- [The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2512.22337/) (2025)
- [Methodical Advice Collection and Reuse in Deep Reinforcement Learning](https://mlsanity.com/paper/2204.07254/) (2022)
- [Ask-AC: An Initiative Advisor-in-the-Loop Actor-Critic Framework](https://mlsanity.com/paper/2207.01955/) (2022)
- [Multi-agent Policy Reciprocity with Theoretical Guarantee](https://mlsanity.com/paper/2304.05632/) (2023)

## Abstract (arXiv)

Collective human knowledge has clearly benefited from the fact that innovations by individuals are taught to others through communication. Similar to human social groups, agents in distributed learning systems would likely benefit from communication to share knowledge and teach skills. The problem of teaching to improve agent learning has been investigated by prior works, but these approaches make assumptions that prevent application of teaching to general multiagent problems, or require domain expertise for problems they can apply to. This learning to teach problem has inherent complexities related to measuring long-term impacts of teaching that compound the standard multiagent coordination challenges. In contrast to existing works, this paper presents the first general framework and algorithm for intelligent agents to learn to teach in a multiagent environment. Our algorithm, Learning to Coordinate and Teach Reinforcement (LeCTR), addresses peer-to-peer teaching in cooperative multiagent reinforcement learning. Each agent in our approach learns both when and what to advise, then uses the received advice to improve local learning. Importantly, these roles are not fixed; these agents learn to assume the role of student and/or teacher at the appropriate moments, requesting and providing advice in order to improve teamwide performance and learning. Empirical comparisons against state-of-the-art teaching methods show that our teaching agents not only learn significantly faster, but also learn to coordinate in tasks where existing methods fail.