# Exploring Computational User Models for Agent Policy Summarization

arXiv: https://arxiv.org/abs/1905.13271 · HTML: https://mlsanity.com/paper/1905.13271/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 30 May 2019.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Built on by (most cited first, 29 of 29)

- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [Counterfactual State Explanations for Reinforcement Learning Agents via Generative Deep Learning](https://mlsanity.com/paper/2101.12446/) (2021)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [Explainable Reinforcement Learning for Broad-XAI: A Conceptual Framework and Survey](https://mlsanity.com/paper/2108.09003/) (2021)
- [Explaining Reinforcement Learning Agents Through Counterfactual Action Outcomes](https://mlsanity.com/paper/2312.11118/) (2023)
- [Integrating Policy Summaries with Reward Decomposition for Explaining Reinforcement Learning Agents](https://mlsanity.com/paper/2210.11825/) (2022)
- ["I Don't Think So": Summarizing Policy Disagreements for Agent Comparison](https://mlsanity.com/paper/2102.03064/) (2021)
- [Feature-Based Interpretable Reinforcement Learning based on State-Transition Models](https://mlsanity.com/paper/2105.07099/) (2021)
- [Reasoning about Counterfactuals to Improve Human Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.01855/) (2022)
- [Local Explanations for Reinforcement Learning](https://mlsanity.com/paper/2202.03597/) (2022)
- [Natural Language Specification of Reinforcement Learning Policies through Differentiable Decision Trees](https://mlsanity.com/paper/2101.07140/) (2021)
- [Explaining Reward Functions to Humans for Better Human-Robot Collaboration](https://mlsanity.com/paper/2110.04192/) (2021)
- [Advantage Actor-Critic with Reasoner: Explaining the Agent's Behavior from an Exploratory Perspective](https://mlsanity.com/paper/2309.04707/) (2023)
- [Towards Interpretable Deep Reinforcement Learning Models via Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.16464/) (2022)
- [A Framework for Understanding and Visualizing Strategies of RL Agents](https://mlsanity.com/paper/2208.08552/) (2022)
- [Towards Reconciling Usability and Usefulness of Explainable AI Methodologies](https://mlsanity.com/paper/2301.05347/) (2023)
- [REACT: Revealing Evolutionary Action Consequence Trajectories for Interpretable Reinforcement Learning](https://mlsanity.com/paper/2404.03359/) (2024)
- [Navigating Trade-offs: Policy Summarization for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2411.04784/) (2024)
- [Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2504.05393/) (2025)
- [(When) Are Contrastive Explanations of Reinforcement Learning Helpful?](https://mlsanity.com/paper/2211.07719/) (2022)
- [I-CEE: Tailoring Explanations of Image Classification Models to User Expertise](https://mlsanity.com/paper/2312.12102/) (2023)
- [Structure and Reduction of MCTS for Explainable-AI](https://mlsanity.com/paper/2408.05488/) (2024)
- [Surrogate Fitness Metrics for Interpretable Reinforcement Learning](https://mlsanity.com/paper/2504.14645/) (2025)
- [Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learning](https://mlsanity.com/paper/2211.06376/) (2022)
- [ASQ-IT: Interactive Explanations for Reinforcement-Learning Agents](https://mlsanity.com/paper/2301.09941/) (2023)
- [IxDRL: A Novel Explainable Deep Reinforcement Learning Toolkit based on Analyses of Interestingness](https://mlsanity.com/paper/2307.08933/) (2023)
- [Closed-loop Teaching via Demonstrations to Improve Policy Transparency](https://mlsanity.com/paper/2406.11850/) (2024)
- ["Trust me on this" Explaining Agent Behavior to a Human Terminator](https://mlsanity.com/paper/2504.04592/) (2025)

## Abstract (arXiv)

AI agents are being developed to support high stakes decision-making processes from driving cars to prescribing drugs, making it increasingly important for human users to understand their behavior. Policy summarization methods aim to convey strengths and weaknesses of such agents by demonstrating their behavior in a subset of informative states. Some policy summarization methods extract a summary that optimizes the ability to reconstruct the agent's policy under the assumption that users will deploy inverse reinforcement learning. In this paper, we explore the use of different models for extracting summaries. We introduce an imitation learning-based approach to policy summarization; we demonstrate through computational simulations that a mismatch between the model used to extract a summary and the model used to reconstruct the policy results in worse reconstruction quality; and we demonstrate through a human-subject study that people use different models to reconstruct policies in different contexts, and that matching the summary extraction model to these can improve performance. Together, our results suggest that it is important to carefully consider user models in policy summarization.