mlsanityData to 14 Oct 2026

The Utility of Explainable AI in Ad Hoc Human-Machine Teaming

arXiv 2209.03943 ↗First posted 8 Sep 2022cs.AI cs.HCcited by 30

Not on the map. Why?

Abstract (arXiv)

Recent advances in machine learning have led to growing interest in Explainable AI (xAI) to enable humans to gain insight into the decision-making of machine learning models. Despite this recent interest, the utility of xAI techniques has not yet been characterized in human-machine teaming. Importantly, xAI offers the promise of enhancing team situational awareness (SA) and shared mental model development, which are the key characteristics of effective human-machine teams. Rapidly developing such mental models is especially critical in ad hoc human-machine teaming, where agents do not have a priori knowledge of others' decision-making strategies. In this paper, we present two novel human-subject experiments quantifying the benefits of deploying xAI techniques within a human-machine teaming scenario. First, we show that xAI techniques can support SA (p<0.05). Second, we examine how different SA levels induced via a collaborative AI policy abstraction affect ad hoc human-machine teaming performance. Importantly, we find that the benefits of xAI are not universal, as there is a strong dependence on the composition of the human-machine team. Novices benefit from xAI providing increased SA (p<0.05) but are susceptible to cognitive overhead (p<0.05). On the other hand, expert performance degrades with the addition of xAI-based support (p<0.05), indicating that the cost of paying attention to the xAI outweighs the benefits obtained from being provided additional information to enhance SA. Our results demonstrate that researchers must deliberately design and deploy the right xAI techniques in the right scenario by carefully considering human-machine team composition and how the xAI method augments SA.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 15 of its 71 references that are in the map. First posted on arXiv 8 Sep 2022.

Builds on

15 of 71 references.

References · 15

  1. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  2. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 399
  3. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  4. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  5. 2018On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detectioncited by 159
  6. 2018Grounding Visual Explanationscited by 93
  7. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  8. 2018Generating Counterfactual Explanations with Natural Languagecited by 37
  9. 2014Efficient Model Learning for Human-Robot Collaborative Taskscited by 35
  10. 2020Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillationcited by 19
  11. 2020Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QAcited by 17
  12. 2019Interpretable and Personalized Apprenticeship Scheduling: Learning Interpretable Scheduling Policies from Heterogeneous User Demonstrationscited by 14
3 more
  1. 2015Decentralized Control of Partially Observable Markov Decision Processes using Belief Space Macro-actionscited by 11
  2. 2021Towards an AI Coach to Infer Team Mental Model Alignment in Healthcarecited by 11
  3. 2021Heterogeneous Graph Attention Networks for Learning Diverse Communicationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 5; 2024: 10; 2025: 6; 2026: 6.

30 papers.

Sort:

By area:

  1. 2022Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanationscited by 85
  2. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  3. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  4. 2024Explaining Bayesian Optimization by Shapley Values Facilitates Human-AI Collaborationcited by 12
  5. 2024Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scarycited by 10
  6. 2023Capturing Humans' Mental Models of AI: An Item Response Theory Approachcited by 8
  7. 2024Human Delegation Behavior in Human-AI Collaboration: The Effect of Contextual Informationcited by 5
  8. 2022Painting the black box white: experimental findings from applying XAI to an ECG reading settingcited by 4
  9. 2023An Information Bottleneck Characterization of the Understanding-Workload Tradeoffcited by 3
  10. 2025Towards Balancing Preference and Performance through Adaptive Personalized Explainabilitycited by 3
  11. 2022Towards Human-Centred Explainability Benchmarks For Text Classificationcited by 2
  12. 2023Towards Reconciling Usability and Usefulness of Explainable AI Methodologiescited by 2
18 more
  1. 2024Mixed-Initiative Human-Robot Teaming under Suboptimality with Online Bayesian Adaptationcited by 2
  2. 2026Event-Grounded Sparse Autoencoders for Vision-Language-Action Policiescited by 2
  3. 2023Modifying RL Policies with Imagined Actions: How Predictable Policies Can Enable Users to Perform Novel Taskscited by 1
  4. 2023Mining multi-modal communication patterns in interaction with explainable and non-explainable robotscited by 1
  5. 2024AIDE: Antithetical, Intent-based, and Diverse Example-Based Explanationscited by 1
  6. 2024Demonstration Based Explainable AI for Learning from Demonstration Methodscited by 1
  7. 2024AI-Driven Human-Autonomy Teaming in Tactical Operations: Proposed Framework, Challenges, and Future Directionscited by 1
  8. 2024Explainable deep learning improves human mental models of self-driving carscited by 1
  9. 2025Enabling Rapid Shared Human-AI Mental Model Alignment via the After-Action Reviewcited by 1
  10. 2025Zero-Shot Coordination in Ad Hoc Teams with Generalized Policy Improvement and Difference Rewardscited by 1
  11. 2024Imagining In-distribution States: How Predictable Robot Behavior Can Enable User Control Over Learned Policiescited by 0
  12. 2025Some Optimizers are More Equal: Understanding the Role of Optimizers in Group Fairnesscited by 0
  13. 2025Adaptive XAI in High Stakes Environments: Modeling Swift Trust with Multimodal Feedback in Human AI Teamscited by 0
  14. 2026Beyond Partner Diversity: An Influence-Based Team Steering Framework for Zero-Shot Human-Machine Teamingcited by 0
  15. 2026Differentiable Belief-based Opponent Shapingcited by 0
  16. 2026Themis: An explainable AI-enabled framework for Reinforcement Learning with Human Feedbackcited by 0
  17. 2026Evaluating XAI Support From A Hierarchical Reinforcement Learning Policy in Human-Agent Collaborationcited by 0
  18. 2026Evaluating RL Explainability Methods by How Much They Help Fix Bugs in Agentscited by 0