mlsanityData to 14 Oct 2026

Explainable Multi-Agent Reinforcement Learning for Temporal Queries

arXiv 2305.10378 ↗First posted 17 May 2023cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

As multi-agent reinforcement learning (MARL) systems are increasingly deployed throughout society, it is imperative yet challenging for users to understand the emergent behaviors of MARL agents in complex environments. This work presents an approach for generating policy-level contrastive explanations for MARL to answer a temporal user query, which specifies a sequence of tasks completed by agents with possible cooperation. The proposed approach encodes the temporal query as a PCTL logic formula and checks if the query is feasible under a given MARL policy via probabilistic model checking. Such explanations can help reconcile discrepancies between the actual and anticipated multi-agent behaviors. The proposed approach also generates correct and complete explanations to pinpoint reasons that make a user query infeasible. We have successfully applied the proposed approach to four benchmark MARL domains (up to 9 agents in one domain). Moreover, the results of a user study show that the generated explanations significantly improve user performance and satisfaction.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 16 of its 41 references that are in the map. First posted on arXiv 17 May 2023.

Builds on

16 of 41 references.

References · 16

  1. 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,206
  2. 2018Metrics for Explainable AI: Challenges and Prospectscited by 260
  3. 2020Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Taskscited by 212
  4. 2019Explainable Reinforcement Learning Through a Causal Lenscited by 177
  5. 2020Explainable Reinforcement Learning: A Surveycited by 141
  6. 2020Shared Experience Actor-Critic for Multi-Agent Reinforcement Learningcited by 116
  7. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  8. 2021Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learningcited by 39
  9. 2019Generation of Policy-Level Explanations for Reinforcement Learningcited by 37
  10. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32
  11. 2021Iterative Bounding MDPs: Learning Interpretable Policies via Non-Interpretable Methodscited by 31
  12. 2021Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Valuescited by 25
4 more
  1. 2020Re-understanding Finite-State Representations of Recurrent Policy Networkscited by 15
  2. 2020Contrastive Explanations for Reinforcement Learning via Embedded Self Predictionscited by 13
  3. 2022Toward Policy Explanations for Multi-Agent Reinforcement Learningcited by 11
  4. 2022Explainable Reinforcement Learning via Model Transformscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 2.

9 papers.

  1. 2024Explaining Decisions of Agents in Mixed-Motive Gamescited by 4
  2. 2024ADESSE: Advice Explanations in Complex Repeated Decision-Making Environmentscited by 3
  3. 2025Integrating Counterfactual Simulations with Language Models for Explaining Multi-Agent Behaviourcited by 3
  4. 2025Learning Probabilistic Temporal Logic Specifications for Stochastic Systemscited by 2
  5. 2024Contrastive Explainable Clustering with Differential Privacycited by 1
  6. 2025Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learningcited by 0
  7. 2025Explaining Decentralized Multi-Agent Reinforcement Learning Policiescited by 0
  8. 2026Interpretable Failure Analysis in Multi-Agent Reinforcement Learning Systemscited by 0
  9. 2026Escaping Local Views: Discovering Latent Concepts for Interpretable Multi-Agent Reinforcement Learningcited by 0