# VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions

arXiv: https://arxiv.org/abs/1803.07464 · HTML: https://mlsanity.com/paper/1803.07464/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Mar 2018.

## Built on by (most cited first, 30 of 50)

- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing](https://mlsanity.com/paper/2102.12060/) (2021)
- [e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks](https://mlsanity.com/paper/2105.03761/) (2021)
- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks](https://mlsanity.com/paper/2203.05081/) (2022)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [CLOVA: A Closed-Loop Visual Assistant with Tool Usage and Update](https://mlsanity.com/paper/2312.10908/) (2023)
- [Visual Instruction Tuning with Polite Flamingo](https://mlsanity.com/paper/2307.01003/) (2023)
- [Local Interpretations for Explainable Natural Language Processing: A Survey](https://mlsanity.com/paper/2103.11072/) (2021)
- [On the Diversity and Limits of Human Explanations](https://mlsanity.com/paper/2106.11988/) (2021)
- [GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis](https://mlsanity.com/paper/2411.16778/) (2024)
- [I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision](https://mlsanity.com/paper/2211.09778/) (2022)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models](https://mlsanity.com/paper/2310.10942/) (2023)
- [CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations](https://mlsanity.com/paper/2204.02380/) (2022)
- [A Survey on Safe Multi-Modal Learning System](https://mlsanity.com/paper/2402.05355/) (2024)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts](https://mlsanity.com/paper/2310.20159/) (2023)
- [MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding](https://mlsanity.com/paper/2502.15786/) (2025)
- [Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective](https://mlsanity.com/paper/2407.02814/) (2024)
- [REX: Reasoning-aware and Grounded Explanation](https://mlsanity.com/paper/2203.06107/) (2022)
- [Uni-NLX: Unifying Textual Explanations for Vision and Vision-Language Tasks](https://mlsanity.com/paper/2308.09033/) (2023)
- [Joint Answering and Explanation for Visual Commonsense Reasoning](https://mlsanity.com/paper/2202.12626/) (2022)
- [CARETS: A Consistency And Robustness Evaluative Test Suite for VQA](https://mlsanity.com/paper/2203.07613/) (2022)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [LLaVA Needs More Knowledge: Retrieval Augmented Natural Language Generation with Knowledge Graph for Explaining Thoracic Pathologies](https://mlsanity.com/paper/2410.04749/) (2024)
- [V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models](https://mlsanity.com/paper/2504.06148/) (2025)
- [ChartQA-X: Generating Explanations for Visual Chart Reasoning](https://mlsanity.com/paper/2504.13275/) (2025)
- [ZARA: Improving Few-Shot Self-Rationalization for Small Language Models](https://mlsanity.com/paper/2305.07355/) (2023)
- [Interactive Question Answering Systems: Literature Review](https://mlsanity.com/paper/2209.01621/) (2022)

## Abstract (arXiv)

Most existing works in visual question answering (VQA) are dedicated to improving the accuracy of predicted answers, while disregarding the explanations. We argue that the explanation for an answer is of the same or even more importance compared with the answer itself, since it makes the question and answering process more understandable and traceable. To this end, we propose a new task of VQA-E (VQA with Explanation), where the computational models are required to generate an explanation with the predicted answer. We first construct a new dataset, and then frame the VQA-E problem in a multi-task learning architecture. Our VQA-E dataset is automatically derived from the VQA v2 dataset by intelligently exploiting the available captions. We have conducted a user study to validate the quality of explanations synthesized by our method. We quantitatively show that the additional supervision from explanations can not only produce insightful textual sentences to justify the answers, but also improve the performance of answer prediction. Our model outperforms the state-of-the-art methods by a clear margin on the VQA v2 dataset.