mlsanityData to 7 Oct 2026

Concept Learning for Interpretable Multi-Agent Reinforcement Learning

arXiv 2302.12232 ↗First posted 23 Feb 2023cs.LG cs.AI cs.ROcited by 18

Not on the map. Why?

Abstract (arXiv)

Multi-agent robotic systems are increasingly operating in real-world environments in close proximity to humans, yet are largely controlled by policy models with inscrutable deep neural network representations. We introduce a method for incorporating interpretable concepts from a domain expert into models trained through multi-agent reinforcement learning, by requiring the model to first predict such concepts then utilize them for decision making. This allows an expert to both reason about the resulting concept policy models in terms of these high-level concepts at run-time, as well as intervene and correct mispredictions to improve performance. We show that this yields improved interpretability and training stability, with benefits to policy performance and sample efficiency in a simulated and real-world cooperative-competitive multi-agent game.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

24 of 33 references.

References · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  3. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  5. 2020Concept Bottleneck Modelscited by 1,022
  6. 2018A Survey Of Methods For Explaining Black Box Modelscited by 970
  7. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  8. 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 318
  9. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  10. 2020Concept Whitening for Interpretable Image Recognitioncited by 225
  11. 2018Local Rule-Based Explanations of Black Box Decision Systemscited by 165
  12. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
12 more
  1. 2018State Representation Learning for Control: An Overviewcited by 125
  2. 2019Iterative Normalization: Beyond Standardization towards Efficient Whiteningcited by 92
  3. 2018Toward Interpretable Deep Reinforcement Learning with Linear Model U-Treescited by 65
  4. 2014Falling Rule Listscited by 58
  5. 2019Interpretability Beyond Classification Output: Semantic Bottleneck Networkscited by 49
  6. 2017Interpretability via Model Extractioncited by 40
  7. 2019Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulationcited by 33
  8. 2020Interpretable Time Series Classification using Linear Models and Multi-resolution Multi-domain Symbolic Representationscited by 31
  9. 2020What Did You Think Would Happen? Explaining Agent Behaviour Through Intended Outcomescited by 20
  10. 2020Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenariocited by 19
  11. 2020Contrastive Explanations for Reinforcement Learning via Embedded Self Predictionscited by 13
  12. 2022Explainable Action Advising for Multi-Agent Reinforcement Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 5; 2025: 4; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
  2. 2023Theory of Mind as Intrinsic Motivation for Multi-Agent Reinforcement Learningcited by 19
  3. 2023Sample-Efficient Learning of Novel Visual Conceptscited by 8
  4. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  5. 2023Benchmarking and Enhancing Disentanglement in Concept-Residual Modelscited by 5
  6. 2024LICORICE: Label-Efficient Concept-Based Interpretable Reinforcement Learningcited by 4
  7. 2025Model-Agnostic Policy Explanations with Large Language Modelscited by 4
  8. 2023Explaining Agent Behavior with Large Language Modelscited by 3
  9. 2023Understanding Your Agent: Leveraging Large Language Models for Behavior Explanationcited by 3
  10. 2025Nonparametric Identification of Latent Conceptscited by 3
  11. 2023Introspective Action Advising for Interpretable Transfer Learningcited by 2
  12. 2024Robot Behavior Personalization from Sparse User Feedbackcited by 2
6 more
  1. 2024Enhancing Interpretability in Deep Reinforcement Learning through Semantic Clusteringcited by 1
  2. 2026Selecting Decision-Relevant Concepts in Reinforcement Learningcited by 1
  3. 2024SNN-Based Online Learning of Concepts and Action Laws in an Open Worldcited by 0
  4. 2025Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learningcited by 0
  5. 2025SALSA-RL: Stability Analysis in the Latent Space of Actions for Reinforcement Learningcited by 0
  6. 2026Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithmscited by 0