# Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Mining

arXiv: https://arxiv.org/abs/1808.00265 · HTML: https://mlsanity.com/paper/1808.00265/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2018.

## Built on by (most cited first, 28 of 28)

- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Grounding Answers for Visual Questions Asked by Visually Impaired People](https://mlsanity.com/paper/2202.01993/) (2022)
- [Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning](https://mlsanity.com/paper/2212.03954/) (2022)
- [Greedy Gradient Ensemble for Robust Visual Question Answering](https://mlsanity.com/paper/2107.12651/) (2021)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering](https://mlsanity.com/paper/2110.01013/) (2021)
- [RES: A Robust Framework for Guiding Visual Explanation](https://mlsanity.com/paper/2206.13413/) (2022)
- [Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery](https://mlsanity.com/paper/2408.04958/) (2024)
- [VQA Therapy: Exploring Answer Differences by Visually Grounding Answers](https://mlsanity.com/paper/2308.11662/) (2023)
- [A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2201.05299/) (2022)
- [Answer Questions with Right Image Regions: A Visual Attention Regularization Approach](https://mlsanity.com/paper/2102.01916/) (2021)
- [Weakly Supervised Grounding for VQA in Vision-Language Transformers](https://mlsanity.com/paper/2207.02334/) (2022)
- [Sentence Attention Blocks for Answer Grounding](https://mlsanity.com/paper/2309.11593/) (2023)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [More Than Just Attention: Improving Cross-Modal Attentions with Contrastive Constraints for Image-Text Matching](https://mlsanity.com/paper/2105.09597/) (2021)
- [COIN: Counterfactual Image Generation for VQA Interpretation](https://mlsanity.com/paper/2201.03342/) (2022)
- [Interpretable Visual Question Answering via Reasoning Supervision](https://mlsanity.com/paper/2309.03726/) (2023)
- [CNN Attention Guidance for Improved Orthopedics Radiographic Fracture Classification](https://mlsanity.com/paper/2203.10690/) (2022)
- [Learning to Agree on Vision Attention for Visual Commonsense Reasoning](https://mlsanity.com/paper/2302.02117/) (2023)
- [Interpreting Vision and Language Generative Models with Semantic Visual Priors](https://mlsanity.com/paper/2304.14986/) (2023)
- [A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models](https://mlsanity.com/paper/2309.02691/) (2023)
- [An experimental study of the vision-bottleneck in VQA](https://mlsanity.com/paper/2202.06858/) (2022)
- [Attention in Reasoning: Dataset, Analysis, and Modeling](https://mlsanity.com/paper/2204.09774/) (2022)
- [Continual Referring Expression Comprehension via Dual Modular Memorization](https://mlsanity.com/paper/2311.14909/) (2023)
- [Efficient Text-Attributed Graph Learning through Selective Annotation and Graph Alignment](https://mlsanity.com/paper/2506.07168/) (2025)
- [Integrating Object Interaction Self-Attention and GAN-Based Debiasing for Visual Question Answering](https://mlsanity.com/paper/2509.20884/) (2025)
- [Calibrating Transformer Attention via Task-Space Sensitivity Feedback](https://mlsanity.com/paper/2512.20661/) (2025)

## Abstract (arXiv)

A key aspect of VQA models that are interpretable is their ability to ground their answers to relevant regions in the image. Current approaches with this capability rely on supervised learning and human annotated groundings to train attention mechanisms inside the VQA architecture. Unfortunately, obtaining human annotations specific for visual grounding is difficult and expensive. In this work, we demonstrate that we can effectively train a VQA architecture with grounding supervision that can be automatically obtained from available region descriptions and object annotations. We also show that our model trained with this mined supervision generates visual groundings that achieve a higher correlation with respect to manually-annotated groundings, meanwhile achieving state-of-the-art VQA accuracy.