# Human Evaluation of Spoken vs. Visual Explanations for Open-Domain QA

arXiv: https://arxiv.org/abs/2012.15075 · HTML: https://mlsanity.com/paper/2012.15075/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2020.

## Built on by (most cited first, 17 of 17)

- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- [Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies](https://mlsanity.com/paper/2112.11471/) (2021)
- [Appropriate Reliance on AI Advice: Conceptualization and the Effect of Explanations](https://mlsanity.com/paper/2302.02187/) (2023)
- [Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making](https://mlsanity.com/paper/2208.07960/) (2022)
- [ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writing](https://mlsanity.com/paper/2305.09770/) (2023)
- [The Utility of Explainable AI in Ad Hoc Human-Machine Teaming](https://mlsanity.com/paper/2209.03943/) (2022)
- [Teaching Humans When To Defer to a Classifier via Exemplars](https://mlsanity.com/paper/2111.11297/) (2021)
- [Selective Explanations: Leveraging Human Input to Align Explainable AI](https://mlsanity.com/paper/2301.09656/) (2023)
- [Should I Follow AI-based Advice? Measuring Appropriate Reliance in Human-AI Decision-Making](https://mlsanity.com/paper/2204.06916/) (2022)
- [A Diachronic Perspective on User Trust in AI under Uncertainty](https://mlsanity.com/paper/2310.13544/) (2023)
- [Machine Explanations and Human Understanding](https://mlsanity.com/paper/2202.04092/) (2022)
- [Effective Human-AI Teams via Learned Natural Language Rules and Onboarding](https://mlsanity.com/paper/2311.01007/) (2023)
- [Generating Full Length Wikipedia Biographies: The Impact of Gender Bias on the Retrieval-Based Generation of Women Biographies](https://mlsanity.com/paper/2204.05879/) (2022)
- [Exploring The Role of Local and Global Explanations in Recommender Systems](https://mlsanity.com/paper/2109.13301/) (2021)
- [Underspecified Human Decision Experiments Considered Harmful](https://mlsanity.com/paper/2401.15106/) (2024)
- [Why is "Problems" Predictive of Positive Sentiment? A Case Study of Explaining Unintuitive Features in Sentiment Classification](https://mlsanity.com/paper/2406.03594/) (2024)
- [Right, No Matter Why: AI Fact-checking and AI Authority in Health-related Inquiry Settings](https://mlsanity.com/paper/2310.14358/) (2023)

## Abstract (arXiv)

While research on explaining predictions of open-domain QA systems (ODQA) to users is gaining momentum, most works have failed to evaluate the extent to which explanations improve user trust. While few works evaluate explanations using user studies, they employ settings that may deviate from the end-user's usage in-the-wild: ODQA is most ubiquitous in voice-assistants, yet current research only evaluates explanations using a visual display, and may erroneously extrapolate conclusions about the most performant explanations to other modalities. To alleviate these issues, we conduct user studies that measure whether explanations help users correctly decide when to accept or reject an ODQA system's answer. Unlike prior work, we control for explanation modality, e.g., whether they are communicated to users through a spoken or visual interface, and contrast effectiveness across modalities. Our results show that explanations derived from retrieved evidence passages can outperform strong baselines (calibrated confidence) across modalities but the best explanation strategy in fact changes with the modality. We show common failure cases of current explanations, emphasize end-to-end evaluation of explanations, and caution against evaluating them in proxy modalities that are different from deployment.