# Improving VQA and its Explanations  by Comparing Competing Explanations

arXiv: https://arxiv.org/abs/2006.15631 · HTML: https://mlsanity.com/paper/2006.15631/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2020.

## Built on by (most cited first, 12 of 12)

- [Multi-Modal Answer Validation for Knowledge-Based VQA](https://mlsanity.com/paper/2103.12248/) (2021)
- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era](https://mlsanity.com/paper/2403.08946/) (2024)
- [Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering](https://mlsanity.com/paper/2103.05568/) (2021)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations](https://mlsanity.com/paper/2204.02380/) (2022)
- [Are Bias Mitigation Techniques for Deep Learning Effective?](https://mlsanity.com/paper/2104.00170/) (2021)
- [REX: Reasoning-aware and Grounded Explanation](https://mlsanity.com/paper/2203.06107/) (2022)
- [Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injection](https://mlsanity.com/paper/2112.06888/) (2021)
- [Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering](https://mlsanity.com/paper/2210.10176/) (2022)
- [II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering](https://mlsanity.com/paper/2402.11058/) (2024)
- [Towards Reasoning-Aware Explainable VQA](https://mlsanity.com/paper/2211.05190/) (2022)

## Abstract (arXiv)

Most recent state-of-the-art Visual Question Answering (VQA) systems are opaque black boxes that are only trained to fit the answer distribution given the question and visual content. As a result, these systems frequently take shortcuts, focusing on simple visual concepts or question priors. This phenomenon becomes more problematic as the questions become complex that requires more reasoning and commonsense knowledge. To address this issue, we present a novel framework that uses explanations for competing answers to help VQA systems select the correct answer. By training on human textual explanations, our framework builds better representations for the questions and visual content, and then reweights confidences in the answer candidates using either generated or retrieved explanations from the training set. We evaluate our framework on the VQA-X dataset, which has more difficult questions with human explanations, achieving new state-of-the-art results on both VQA and its explanations.