# Explaining Decisions of Agents in Mixed-Motive Games

arXiv: https://arxiv.org/abs/2407.15255 · HTML: https://mlsanity.com/paper/2407.15255/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 21 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Playing repeated games with Large Language Models](https://mlsanity.com/paper/2305.16867/) (2023)
- [Explainability in Deep Reinforcement Learning](https://mlsanity.com/paper/2008.06693/) (2020)
- [Contrastive Explanation: A Structural-Model Approach](https://mlsanity.com/paper/1811.03163/) (2018)
- [Modeling Strong and Human-Like Gameplay with KL-Regularized Search](https://mlsanity.com/paper/2112.07544/) (2021)
- [No Press Diplomacy: Modeling Multi-Agent Gameplay](https://mlsanity.com/paper/1909.02128/) (2019)
- [No-Press Diplomacy from Scratch](https://mlsanity.com/paper/2110.02924/) (2021)
- [Reviewing the Need for Explainable Artificial Intelligence (xAI)](https://mlsanity.com/paper/2012.01007/) (2020)
- [Learning to Play No-Press Diplomacy with Best Response Policy Iteration](https://mlsanity.com/paper/2006.04635/) (2020)
- [Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values](https://mlsanity.com/paper/2110.01307/) (2021)
- [LARP: Language-Agent Role Play for Open-World Games](https://mlsanity.com/paper/2312.17653/) (2023)
- [A critical analysis of metrics used for measuring progress in artificial intelligence](https://mlsanity.com/paper/2008.02577/) (2020)
- [AI for Explaining Decisions in Multi-Agent Environments](https://mlsanity.com/paper/1910.04404/) (2019)
- [MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.12449/) (2022)
- [Toward Policy Explanations for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2204.12568/) (2022)
- [Explainable Action Advising for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2211.07882/) (2022)
- [Explainable Multi-Agent Reinforcement Learning for Temporal Queries](https://mlsanity.com/paper/2305.10378/) (2023)
- [Efficiently Quantifying Individual Agent Importance in Cooperative MARL](https://mlsanity.com/paper/2312.08466/) (2023)
- [`Why didn't you allocate this task to them?' Negotiation-Aware Explicable Task Allocation and Contrastive Explanation Generation](https://mlsanity.com/paper/2002.01640/) (2020)
- [On Diagnostics for Understanding Agent Training Behaviour in Cooperative MARL](https://mlsanity.com/paper/2312.08468/) (2023)
- [Towards a more efficient computation of individual attribute and policy contribution for post-hoc explanation of cooperative multi-agent systems using Myerson values](https://mlsanity.com/paper/2212.03041/) (2022)

## Built on by (most cited first, 4 of 4)

- [A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarios](https://mlsanity.com/paper/2412.03920/) (2024)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [Game-Theoretic Lens on LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2601.15047/) (2026)
- [Learning Incentive Structures for Cooperative Resilience in Multi-Agent Systems under Social Dilemmas](https://mlsanity.com/paper/2601.22292/) (2026)

## Abstract (arXiv)

In recent years, agents have become capable of communicating seamlessly via natural language and navigating in environments that involve cooperation and competition, a fact that can introduce social dilemmas. Due to the interleaving of cooperation and competition, understanding agents' decision-making in such environments is challenging, and humans can benefit from obtaining explanations. However, such environments and scenarios have rarely been explored in the context of explainable AI. While some explanation methods for cooperative environments can be applied in mixed-motive setups, they do not address inter-agent competition, cheap-talk, or implicit communication by actions. In this work, we design explanation methods to address these issues. Then, we proceed to establish generality and demonstrate the applicability of the methods to three games with vastly different properties. Lastly, we demonstrate the effectiveness and usefulness of the methods for humans in two mixed-motive games. The first is a challenging 7-player game called no-press Diplomacy. The second is a 3-player game inspired by the prisoner's dilemma, featuring communication in natural language.