# "I Don't Think So": Summarizing Policy Disagreements for Agent Comparison

arXiv: https://arxiv.org/abs/2102.03064 · HTML: https://mlsanity.com/paper/2102.03064/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 20 references that are in the map.
- First posted on arXiv 5 Feb 2021.

## Builds on (most cited first, 9 of 9)

- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Metrics for Explainable AI: Challenges and Prospects](https://mlsanity.com/paper/1812.04608/) (2018)
- [Visualizing and Understanding Atari Agents](https://mlsanity.com/paper/1711.00138/) (2017)
- [Explainable Reinforcement Learning: A Survey](https://mlsanity.com/paper/2005.06247/) (2020)
- [Explainability in Deep Reinforcement Learning](https://mlsanity.com/paper/2008.06693/) (2020)
- [Interestingness Elements for Explainable Reinforcement Learning: Understanding Agents' Capabilities and Limitations](https://mlsanity.com/paper/1912.09007/) (2019)
- [Enabling Robots to Communicate their Objectives](https://mlsanity.com/paper/1702.03465/) (2017)
- [Local and Global Explanations of Agent Behavior: Integrating Strategy Summaries with Saliency Maps](https://mlsanity.com/paper/2005.08874/) (2020)
- [Exploring Computational User Models for Agent Policy Summarization](https://mlsanity.com/paper/1905.13271/) (2019)

## Built on by (most cited first, 6 of 6)

- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes](https://mlsanity.com/paper/2312.11118/) (2023)
- [Enhancing RL Safety with Counterfactual LLM Reasoning](https://mlsanity.com/paper/2409.10188/) (2024)
- [A Survey on Explainable Reinforcement Learning: Concepts, Algorithms, Challenges](https://mlsanity.com/paper/2211.06665/) (2022)
- [(When) Are Contrastive Explanations of Reinforcement Learning Helpful?](https://mlsanity.com/paper/2211.07719/) (2022)
- [Conveying Autonomous Robot Capabilities through Contrasting Behaviour Summaries](https://mlsanity.com/paper/2304.00367/) (2023)

## Abstract (arXiv)

With Artificial Intelligence on the rise, human interaction with autonomous agents becomes more frequent. Effective human-agent collaboration requires users to understand the agent's behavior, as failing to do so may cause reduced productivity, misuse or frustration. Agent strategy summarization methods are used to describe the strategy of an agent to its destined user through demonstration. A summary's objective is to maximize the user's understanding of the agent's aptitude by showcasing its behaviour in a selected set of world states. While shown to be useful, we show that current methods are limited when tasked with comparing between agents, as each summary is independently generated for a specific agent. In this paper, we propose a novel method for generating dependent and contrastive summaries that emphasize the differences between agent policies by identifying states in which the agents disagree on the best course of action. We conduct user studies to assess the usefulness of disagreement-based summaries for identifying superior agents and conveying agent differences. Results show disagreement-based summaries lead to improved user performance compared to summaries generated using HIGHLIGHTS, a strategy summarization algorithm which generates summaries for each agent independently.