# Visual Grounding Methods for VQA are Working for the Wrong Reasons!

arXiv: https://arxiv.org/abs/2004.05704 · HTML: https://mlsanity.com/paper/2004.05704/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2020.

## Built on by (most cited first, 18 of 18)

- [Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering](https://mlsanity.com/paper/2104.03149/) (2021)
- [Grounding 'Grounding' in NLP](https://mlsanity.com/paper/2106.02192/) (2021)
- [Greedy Gradient Ensemble for Robust Visual Question Answering](https://mlsanity.com/paper/2107.12651/) (2021)
- [VisQA: X-raying Vision and Language Reasoning in Transformers](https://mlsanity.com/paper/2104.00926/) (2021)
- [LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering](https://mlsanity.com/paper/2105.14300/) (2021)
- [How Transferable are Reasoning Patterns in VQA?](https://mlsanity.com/paper/2104.03656/) (2021)
- [Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering](https://mlsanity.com/paper/2109.06122/) (2021)
- [VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives](https://mlsanity.com/paper/2206.11212/) (2022)
- [Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systems](https://mlsanity.com/paper/2103.03048/) (2021)
- [Answer Questions with Right Image Regions: A Visual Attention Regularization Approach](https://mlsanity.com/paper/2102.01916/) (2021)
- [Uncovering the Full Potential of Visual Grounding Methods in VQA](https://mlsanity.com/paper/2401.07803/) (2024)
- [Delving Deeper into Cross-lingual Visual Question Answering](https://mlsanity.com/paper/2202.07630/) (2022)
- [Measuring Faithful and Plausible Visual Grounding in VQA](https://mlsanity.com/paper/2305.15015/) (2023)
- [Supervising the Transfer of Reasoning Patterns in VQA](https://mlsanity.com/paper/2106.05597/) (2021)
- [Guiding Visual Question Answering with Attention Priors](https://mlsanity.com/paper/2205.12616/) (2022)
- [Language bias in Visual Question Answering: A Survey and Taxonomy](https://mlsanity.com/paper/2111.08531/) (2021)
- [Visually Grounded VQA by Lattice-based Retrieval](https://mlsanity.com/paper/2211.08086/) (2022)
- [An Empirical Study on the Language Modal in Visual Question Answering](https://mlsanity.com/paper/2305.10143/) (2023)

## Abstract (arXiv)

Existing Visual Question Answering (VQA) methods tend to exploit dataset biases and spurious statistical correlations, instead of producing right answers for the right reasons. To address this issue, recent bias mitigation methods for VQA propose to incorporate visual cues (e.g., human attention maps) to better ground the VQA models, showcasing impressive gains. However, we show that the performance improvements are not a result of improved visual grounding, but a regularization effect which prevents over-fitting to linguistic priors. For instance, we find that it is not actually necessary to provide proper, human-based cues; random, insensible cues also result in similar improvements. Based on this observation, we propose a simpler regularization scheme that does not require any external annotations and yet achieves near state-of-the-art performance on VQA-CPv2.