# Adversarial Regularization for Visual Question Answering: Strengths, Shortcomings, and Side Effects

arXiv: https://arxiv.org/abs/1906.08430 · HTML: https://mlsanity.com/paper/1906.08430/

## Key facts

- Cited by 25 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 20 Jun 2019.

## Built on by (most cited first, 25 of 25)

- [Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA](https://mlsanity.com/paper/2210.04692/) (2022)
- [End-to-End Self-Debiasing Framework for Robust NLU Training](https://mlsanity.com/paper/2109.02071/) (2021)
- [Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering](https://mlsanity.com/paper/2110.01013/) (2021)
- [BLIND: Bias Removal With No Demographics](https://mlsanity.com/paper/2212.10563/) (2022)
- [Rethinking Data Augmentation for Robust Visual Question Answering](https://mlsanity.com/paper/2207.08739/) (2022)
- [LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering](https://mlsanity.com/paper/2105.14300/) (2021)
- [Check It Again: Progressive Visual Question Answering via Visual Entailment](https://mlsanity.com/paper/2106.04605/) (2021)
- [Are Bias Mitigation Techniques for Deep Learning Effective?](https://mlsanity.com/paper/2104.00170/) (2021)
- [Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answering](https://mlsanity.com/paper/2109.06122/) (2021)
- [Robustifying Multi-hop QA through Pseudo-Evidentiality Training](https://mlsanity.com/paper/2107.03242/) (2021)
- [Towards Debiasing Temporal Sentence Grounding in Video](https://mlsanity.com/paper/2111.04321/) (2021)
- [A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach](https://mlsanity.com/paper/2203.05243/) (2022)
- [OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses](https://mlsanity.com/paper/2204.02426/) (2022)
- [Mitigating the Position Bias of Transformer Models in Passage Re-Ranking](https://mlsanity.com/paper/2101.06980/) (2021)
- [X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering](https://mlsanity.com/paper/2107.11576/) (2021)
- [Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning](https://mlsanity.com/paper/2210.04563/) (2022)
- [Language bias in Visual Question Answering: A Survey and Taxonomy](https://mlsanity.com/paper/2111.08531/) (2021)
- [Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs](https://mlsanity.com/paper/2607.03647/) (2026)
- [An Empirical Study on the Language Modal in Visual Question Answering](https://mlsanity.com/paper/2305.10143/) (2023)
- [Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention](https://mlsanity.com/paper/2410.10184/) (2024)
- [Investigating Ensemble Methods for Model Robustness Improvement of Text Classifiers](https://mlsanity.com/paper/2210.16298/) (2022)
- [Using Focal Loss to Fight Shallow Heuristics: An Empirical Analysis of Modulated Cross-Entropy in Natural Language Inference](https://mlsanity.com/paper/2211.13331/) (2022)
- [Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering](https://mlsanity.com/paper/2210.14558/) (2022)
- [From text to multimodal: a survey of adversarial example generation in question answering systems](https://mlsanity.com/paper/2312.16156/) (2023)
- [Strong denoising of financial time-series](https://mlsanity.com/paper/2408.05690/) (2024)

## Abstract (arXiv)

Visual question answering (VQA) models have been shown to over-rely on linguistic biases in VQA datasets, answering questions "blindly" without considering visual context. Adversarial regularization (AdvReg) aims to address this issue via an adversary sub-network that encourages the main model to learn a bias-free representation of the question. In this work, we investigate the strengths and shortcomings of AdvReg with the goal of better understanding how it affects inference in VQA models. Despite achieving a new state-of-the-art on VQA-CP, we find that AdvReg yields several undesirable side-effects, including unstable gradients and sharply reduced performance on in-domain examples. We demonstrate that gradual introduction of regularization during training helps to alleviate, but not completely solve, these issues. Through error analyses, we observe that AdvReg improves generalization to binary questions, but impairs performance on questions with heterogeneous answer distributions. Qualitatively, we also find that regularized models tend to over-rely on visual features, while ignoring important linguistic cues in the question. Our results suggest that AdvReg requires further refinement before it can be considered a viable bias mitigation technique for VQA.