# A Study on Multimodal and Interactive Explanations for Visual Question Answering

arXiv: https://arxiv.org/abs/2003.00431 · HTML: https://mlsanity.com/paper/2003.00431/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2020.

## Built on by (most cited first, 14 of 14)

- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Interactive Question Answering Systems: Literature Review](https://mlsanity.com/paper/2209.01621/) (2022)
- [How explainable AI affects human performance: A systematic review of the behavioural consequences of saliency maps](https://mlsanity.com/paper/2404.16042/) (2024)
- [Improving Users' Mental Model with Attention-directed Counterfactual Edits](https://mlsanity.com/paper/2110.06863/) (2021)
- [Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models](https://mlsanity.com/paper/2103.14712/) (2021)
- [Interpretable multimodal sentiment analysis based on textual modality descriptions by using large-scale language models](https://mlsanity.com/paper/2305.06162/) (2023)
- [Explainable Interface for Human-Autonomy Teaming: A Survey](https://mlsanity.com/paper/2405.02583/) (2024)
- [Multimodal Coherent Explanation Generation of Robot Failures](https://mlsanity.com/paper/2410.00659/) (2024)
- [Explainable AI the Latest Advancements and New Trends](https://mlsanity.com/paper/2505.07005/) (2025)
- [Knowledge-Based Counterfactual Queries for Visual Question Answering](https://mlsanity.com/paper/2303.02601/) (2023)
- [Who Benefits from AI Explanations? Towards Accessible and Interpretable Systems](https://mlsanity.com/paper/2508.10806/) (2025)

## Abstract (arXiv)

Explainability and interpretability of AI models is an essential factor affecting the safety of AI. While various explainable AI (XAI) approaches aim at mitigating the lack of transparency in deep networks, the evidence of the effectiveness of these approaches in improving usability, trust, and understanding of AI systems are still missing. We evaluate multimodal explanations in the setting of a Visual Question Answering (VQA) task, by asking users to predict the response accuracy of a VQA agent with and without explanations. We use between-subjects and within-subjects experiments to probe explanation effectiveness in terms of improving user prediction accuracy, confidence, and reliance, among other factors. The results indicate that the explanations help improve human prediction accuracy, especially in trials when the VQA system's answer is inaccurate. Furthermore, we introduce active attention, a novel method for evaluating causal attentional effects through intervention by editing attention maps. User explanation ratings are strongly correlated with human prediction accuracy and suggest the efficacy of these explanations in human-machine AI collaboration tasks.