# The Utility of Explainable AI in Ad Hoc Human-Machine Teaming

arXiv: https://arxiv.org/abs/2209.03943 · HTML: https://mlsanity.com/paper/2209.03943/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 71 references that are in the map.
- First posted on arXiv 8 Sep 2022.

## Builds on (most cited first, 15 of 15)

- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making](https://mlsanity.com/paper/2001.02114/) (2020)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [Grounding Visual Explanations](https://mlsanity.com/paper/1807.09685/) (2018)
- [Robot Learning in Homes: Improving Generalization and Reducing Dataset Bias](https://mlsanity.com/paper/1807.07049/) (2018)
- [Generating Counterfactual Explanations with Natural Language](https://mlsanity.com/paper/1806.09809/) (2018)
- [Efficient Model Learning for Human-Robot Collaborative Tasks](https://mlsanity.com/paper/1405.6341/) (2014)
- [Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillation](https://mlsanity.com/paper/2001.00503/) (2020)
- [Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA](https://mlsanity.com/paper/2012.15075/) (2020)
- [Interpretable and Personalized Apprenticeship Scheduling: Learning Interpretable Scheduling Policies from Heterogeneous User Demonstrations](https://mlsanity.com/paper/1906.06397/) (2019)
- [Decentralized Control of Partially Observable Markov Decision Processes using Belief Space Macro-actions](https://mlsanity.com/paper/1502.06030/) (2015)
- [Towards an AI Coach to Infer Team Mental Model Alignment in Healthcare](https://mlsanity.com/paper/2102.08507/) (2021)
- [Heterogeneous Graph Attention Networks for Learning Diverse Communication](https://mlsanity.com/paper/2108.09568/) (2021)

## Built on by (most cited first, 30 of 30)

- [Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations](https://mlsanity.com/paper/2210.11584/) (2022)
- [From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered](https://mlsanity.com/paper/2506.07461/) (2025)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Explaining Bayesian Optimization by Shapley Values Facilitates Human-AI Collaboration](https://mlsanity.com/paper/2403.04629/) (2024)
- [Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary](https://mlsanity.com/paper/2411.10461/) (2024)
- [Capturing Humans' Mental Models of AI: An Item Response Theory Approach](https://mlsanity.com/paper/2305.09064/) (2023)
- [Human Delegation Behavior in Human-AI Collaboration: The Effect of Contextual Information](https://mlsanity.com/paper/2401.04729/) (2024)
- [Painting the black box white: experimental findings from applying XAI to an ECG reading setting](https://mlsanity.com/paper/2210.15236/) (2022)
- [An Information Bottleneck Characterization of the Understanding-Workload Tradeoff](https://mlsanity.com/paper/2310.07802/) (2023)
- [Towards Balancing Preference and Performance through Adaptive Personalized Explainability](https://mlsanity.com/paper/2504.13856/) (2025)
- [Towards Human-Centred Explainability Benchmarks For Text Classification](https://mlsanity.com/paper/2211.05452/) (2022)
- [Towards Reconciling Usability and Usefulness of Explainable AI Methodologies](https://mlsanity.com/paper/2301.05347/) (2023)
- [Mixed-Initiative Human-Robot Teaming under Suboptimality with Online Bayesian Adaptation](https://mlsanity.com/paper/2403.16178/) (2024)
- [Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies](https://mlsanity.com/paper/2605.17204/) (2026)
- [Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Tasks](https://mlsanity.com/paper/2312.05991/) (2023)
- [Mining multi-modal communication patterns in interaction with explainable and non-explainable robots](https://mlsanity.com/paper/2312.14634/) (2023)
- [AIDE: Antithetical, Intent-based, and Diverse Example-Based Explanations](https://mlsanity.com/paper/2407.16010/) (2024)
- [Demonstration Based Explainable AI for Learning from Demonstration Methods](https://mlsanity.com/paper/2410.05715/) (2024)
- [AI-Driven Human-Autonomy Teaming in Tactical Operations: Proposed Framework, Challenges, and Future Directions](https://mlsanity.com/paper/2411.09788/) (2024)
- [Explainable deep learning improves human mental models of self-driving cars](https://mlsanity.com/paper/2411.18714/) (2024)
- [Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Review](https://mlsanity.com/paper/2503.19607/) (2025)
- [Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewards](https://mlsanity.com/paper/2510.16187/) (2025)
- [Imagining In-distribution States: How Predictable Robot Behavior Can Enable User Control Over Learned Policies](https://mlsanity.com/paper/2406.13711/) (2024)
- [Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairness](https://mlsanity.com/paper/2504.14882/) (2025)
- [Adaptive XAI in High Stakes Environments: Modeling Swift Trust with Multimodal Feedback in Human AI Teams](https://mlsanity.com/paper/2507.21158/) (2025)
- [Beyond Partner Diversity: An Influence-Based Team Steering Framework for Zero-Shot Human-Machine Teaming](https://mlsanity.com/paper/2605.15400/) (2026)
- [Differentiable Belief-based Opponent Shaping](https://mlsanity.com/paper/2605.29042/) (2026)
- [Themis: An explainable AI-enabled framework for Reinforcement Learning with Human Feedback](https://mlsanity.com/paper/2606.24622/) (2026)
- [Evaluating XAI Support From A Hierarchical Reinforcement Learning Policy in Human-Agent Collaboration](https://mlsanity.com/paper/2608.06381/) (2026)
- [Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agents](https://mlsanity.com/paper/2608.17524/) (2026)

## Abstract (arXiv)

Recent advances in machine learning have led to growing interest in Explainable AI (xAI) to enable humans to gain insight into the decision-making of machine learning models. Despite this recent interest, the utility of xAI techniques has not yet been characterized in human-machine teaming. Importantly, xAI offers the promise of enhancing team situational awareness (SA) and shared mental model development, which are the key characteristics of effective human-machine teams. Rapidly developing such mental models is especially critical in ad hoc human-machine teaming, where agents do not have a priori knowledge of others' decision-making strategies. In this paper, we present two novel human-subject experiments quantifying the benefits of deploying xAI techniques within a human-machine teaming scenario. First, we show that xAI techniques can support SA (p<0.05). Second, we examine how different SA levels induced via a collaborative AI policy abstraction affect ad hoc human-machine teaming performance. Importantly, we find that the benefits of xAI are not universal, as there is a strong dependence on the composition of the human-machine team. Novices benefit from xAI providing increased SA (p<0.05) but are susceptible to cognitive overhead (p<0.05). On the other hand, expert performance degrades with the addition of xAI-based support (p<0.05), indicating that the cost of paying attention to the xAI outweighs the benefits obtained from being provided additional information to enhance SA. Our results demonstrate that researchers must deliberately design and deploy the right xAI techniques in the right scenario by carefully considering human-machine team composition and how the xAI method augments SA.