# Self-Critical Reasoning for Robust Visual Question Answering

arXiv: https://arxiv.org/abs/1905.09998 · HTML: https://mlsanity.com/paper/1905.09998/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 30 of 61)

- [A Review on Explainability in Multimodal Deep Neural Nets](https://mlsanity.com/paper/2105.07878/) (2021)
- [SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answering](https://mlsanity.com/paper/2204.02285/) (2022)
- [Going Beyond XAI: A Systematic Survey for Explanation-Guided Learning](https://mlsanity.com/paper/2212.03954/) (2022)
- [Greedy Gradient Ensemble for Robust Visual Question Answering](https://mlsanity.com/paper/2107.12651/) (2021)
- [Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA](https://mlsanity.com/paper/2210.04692/) (2022)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [Introspective Distillation for Robust Question Answering](https://mlsanity.com/paper/2111.01026/) (2021)
- [Separating Skills and Concepts for Novel Visual Question Answering](https://mlsanity.com/paper/2107.09106/) (2021)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering](https://mlsanity.com/paper/2110.01013/) (2021)
- [Debiased Fine-Tuning for Vision-language Models by Prompt Regularization](https://mlsanity.com/paper/2301.12429/) (2023)
- [Multimodal Integration of Human-Like Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13139/) (2021)
- [On Modality Bias Recognition and Reduction](https://mlsanity.com/paper/2202.12690/) (2022)
- [AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss](https://mlsanity.com/paper/2105.01993/) (2021)
- [Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem](https://mlsanity.com/paper/2207.11850/) (2022)
- [Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering](https://mlsanity.com/paper/2404.12020/) (2024)
- [LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering](https://mlsanity.com/paper/2105.14300/) (2021)
- [Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instances](https://mlsanity.com/paper/2209.08529/) (2022)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering](https://mlsanity.com/paper/2109.06122/) (2021)
- [REX: Reasoning-aware and Grounded Explanation](https://mlsanity.com/paper/2203.06107/) (2022)
- [VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives](https://mlsanity.com/paper/2206.11212/) (2022)
- [Generative Bias for Robust Visual Question Answering](https://mlsanity.com/paper/2208.00690/) (2022)
- [Unveiling and Mitigating Bias in Audio Visual Segmentation](https://mlsanity.com/paper/2407.16638/) (2024)
- [X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering](https://mlsanity.com/paper/2107.11576/) (2021)
- [S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning](https://mlsanity.com/paper/2309.02155/) (2023)
- [Answer Questions with Right Image Regions: A Visual Attention Regularization Approach](https://mlsanity.com/paper/2102.01916/) (2021)
- [Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning](https://mlsanity.com/paper/2210.04563/) (2022)
- [Uncovering the Full Potential of Visual Grounding Methods in VQA](https://mlsanity.com/paper/2401.07803/) (2024)
- [Measuring Faithful and Plausible Visual Grounding in VQA](https://mlsanity.com/paper/2305.15015/) (2023)

## Abstract (arXiv)

Visual Question Answering (VQA) deep-learning systems tend to capture superficial statistical correlations in the training data because of strong language priors and fail to generalize to test data with a significantly different question-answer (QA) distribution. To address this issue, we introduce a self-critical training objective that ensures that visual explanations of correct answers match the most influential image regions more than other competitive answer candidates. The influential regions are either determined from human visual/textual explanations or automatically from just significant words in the question and answer. We evaluate our approach on the VQA generalization task using the VQA-CP dataset, achieving a new state-of-the-art i.e., 49.5% using textual explanations and 48.5% using automatically annotated regions.