# Explainable Action Advising for Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2211.07882 · HTML: https://mlsanity.com/paper/2211.07882/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 14 of its 35 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 14 of 14)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Verifiable Reinforcement Learning via Policy Extraction](https://mlsanity.com/paper/1805.08328/) (2018)
- [Reinforcement Learning from Imperfect Demonstrations](https://mlsanity.com/paper/1802.05313/) (2018)
- [Learning to Teach in Cooperative Multiagent Reinforcement Learning](https://mlsanity.com/paper/1805.07830/) (2018)
- [Teaching on a Budget in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1905.01357/) (2019)
- [HAMMER: Multi-Level Coordination of Reinforcement Learning Agents via Learned Messaging](https://mlsanity.com/paper/2102.00824/) (2021)
- [An Efficient Transfer Learning Framework for Multiagent Reinforcement Learning](https://mlsanity.com/paper/2002.08030/) (2020)
- [Learning to Teach Reinforcement Learning Agents](https://mlsanity.com/paper/1707.09079/) (2017)
- [Policy Distillation and Value Matching in Multiagent Reinforcement Learning](https://mlsanity.com/paper/1903.06592/) (2019)
- [Action Advising with Advice Imitation in Deep Reinforcement Learning](https://mlsanity.com/paper/2104.08441/) (2021)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)

## Built on by (most cited first, 10 of 10)

- [Concept Learning for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.12232/) (2023)
- [Sample-Efficient Learning of Novel Visual Concepts](https://mlsanity.com/paper/2306.09482/) (2023)
- [Explaining Decisions of Agents in Mixed-Motive Games](https://mlsanity.com/paper/2407.15255/) (2024)
- [Model-Agnostic Policy Explanations with Large Language Models](https://mlsanity.com/paper/2504.05625/) (2025)
- [Explaining Agent Behavior with Large Language Models](https://mlsanity.com/paper/2309.10346/) (2023)
- [Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation](https://mlsanity.com/paper/2311.18062/) (2023)
- [Introspective Action Advising for Interpretable Transfer Learning](https://mlsanity.com/paper/2306.12314/) (2023)
- [Energy-Based Transfer for Reinforcement Learning](https://mlsanity.com/paper/2506.16590/) (2025)
- [Human-Allied Relational Reinforcement Learning](https://mlsanity.com/paper/2510.16188/) (2025)
- [Fidelity-Induced Interpretable Policy Extraction for Reinforcement Learning](https://mlsanity.com/paper/2309.06097/) (2023)

## Abstract (arXiv)

Action advising is a knowledge transfer technique for reinforcement learning based on the teacher-student paradigm. An expert teacher provides advice to a student during training in order to improve the student's sample efficiency and policy performance. Such advice is commonly given in the form of state-action pairs. However, it makes it difficult for the student to reason with and apply to novel states. We introduce Explainable Action Advising, in which the teacher provides action advice as well as associated explanations indicating why the action was chosen. This allows the student to self-reflect on what it has learned, enabling advice generalization and leading to improved sample efficiency and learning performance - even in environments where the teacher is sub-optimal. We empirically show that our framework is effective in both single-agent and multi-agent scenarios, yielding improved policy returns and convergence rates when compared to state-of-the-art methods