# Towards Transparent AI Systems: Interpreting Visual Question Answering Models

arXiv: https://arxiv.org/abs/1608.08974 · HTML: https://mlsanity.com/paper/1608.08974/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2016.

## Built on by (most cited first, 19 of 19)

- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models](https://mlsanity.com/paper/2404.10237/) (2024)
- [MultiViz: Towards Visualizing and Understanding Multimodal Models](https://mlsanity.com/paper/2207.00056/) (2022)
- [DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations](https://mlsanity.com/paper/2203.02013/) (2022)
- [On Explaining Multimodal Hateful Meme Detection Models](https://mlsanity.com/paper/2204.01734/) (2022)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data](https://mlsanity.com/paper/2311.09115/) (2023)
- [Levels of explainable artificial intelligence for human-aligned conversational explanations](https://mlsanity.com/paper/2107.03178/) (2021)
- [VisQA: X-raying Vision and Language Reasoning in Transformers](https://mlsanity.com/paper/2104.00926/) (2021)
- [Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models](https://mlsanity.com/paper/2404.04514/) (2024)
- [A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations](https://mlsanity.com/paper/2105.02626/) (2021)
- [On the Efficacy of Co-Attention Transformer Layers in Visual Question Answering](https://mlsanity.com/paper/2201.03965/) (2022)
- [BinaryVQA: A Versatile Test Set to Evaluate the Out-of-Distribution Generalization of VQA Models](https://mlsanity.com/paper/2301.12032/) (2023)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [Information-theoretic Evolution of Model Agnostic Global Explanations](https://mlsanity.com/paper/2105.06956/) (2021)
- [Advancing Multimodal Fusion on Heterogeneous Medical Data with Hybrid Geometry Attention](https://mlsanity.com/paper/2607.19086/) (2026)

## Abstract (arXiv)

Deep neural networks have shown striking progress and obtained state-of-the-art results in many AI research fields in the recent years. However, it is often unsatisfying to not know why they predict what they do. In this paper, we address the problem of interpreting Visual Question Answering (VQA) models. Specifically, we are interested in finding what part of the input (pixels in images or words in questions) the VQA model focuses on while answering the question. To tackle this problem, we use two visualization techniques -- guided backpropagation and occlusion -- to find important words in the question and important regions in the image. We then present qualitative and quantitative analyses of these importance maps. We found that even without explicit attention mechanisms, VQA models may sometimes be implicitly attending to relevant regions in the image, and often to appropriate words in the question.