# AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss

arXiv: https://arxiv.org/abs/2105.01993 · HTML: https://mlsanity.com/paper/2105.01993/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 30 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [FVQA: Fact-based Visual Question Answering](https://mlsanity.com/paper/1606.05433/) (2016)
- [Counterfactual Samples Synthesizing for Robust Visual Question Answering](https://mlsanity.com/paper/2003.06576/) (2020)
- [Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded](https://mlsanity.com/paper/1902.03751/) (2019)
- [Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering](https://mlsanity.com/paper/1708.01471/) (2017)
- [Overcoming Language Priors in Visual Question Answering with Adversarial Regularization](https://mlsanity.com/paper/1810.03649/) (2018)
- [Self-Critical Reasoning for Robust Visual Question Answering](https://mlsanity.com/paper/1905.09998/) (2019)
- [Ask Your Neurons: A Neural-based Approach to Answering Questions about Images](https://mlsanity.com/paper/1505.01121/) (2015)
- [Learning to Count Objects in Natural Images for Visual Question Answering](https://mlsanity.com/paper/1802.05766/) (2018)
- [Loss re-scaling VQA: Revisiting the LanguagePrior Problem from a Class-imbalance View](https://mlsanity.com/paper/2010.16010/) (2020)
- [Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder](https://mlsanity.com/paper/2007.06198/) (2020)
- [Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering](https://mlsanity.com/paper/1704.03162/) (2017)
- [Learning Visual Question Answering by Bootstrapping Hard Attention](https://mlsanity.com/paper/1808.00300/) (2018)
- [Quantifying and Alleviating the Language Prior Problem in Visual Question Answering](https://mlsanity.com/paper/1905.04877/) (2019)

## Built on by (most cited first, 18 of 18)

- [Voice-Face Homogeneity Tells Deepfake](https://mlsanity.com/paper/2203.02195/) (2022)
- [Temporal Feature Alignment and Mutual Information Maximization for Video-Based Human Pose Estimation](https://mlsanity.com/paper/2203.15227/) (2022)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [2D Human Pose Estimation: A Survey](https://mlsanity.com/paper/2204.07370/) (2022)
- [On Modality Bias Recognition and Reduction](https://mlsanity.com/paper/2202.12690/) (2022)
- [Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem](https://mlsanity.com/paper/2207.11850/) (2022)
- [Joint Answering and Explanation for Visual Commonsense Reasoning](https://mlsanity.com/paper/2202.12626/) (2022)
- [Mitigating Modal Imbalance in Multimodal Reasoning](https://mlsanity.com/paper/2510.02608/) (2025)
- [REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training](https://mlsanity.com/paper/2105.04201/) (2021)
- [Language bias in Visual Question Answering: A Survey and Taxonomy](https://mlsanity.com/paper/2111.08531/) (2021)
- [Do Vision-Language Transformers Exhibit Visual Commonsense? An Empirical Study of VCR](https://mlsanity.com/paper/2405.16934/) (2024)
- [An Empirical Study on the Language Modal in Visual Question Answering](https://mlsanity.com/paper/2305.10143/) (2023)
- [VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Tasks](https://mlsanity.com/paper/2407.19795/) (2024)
- [Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention](https://mlsanity.com/paper/2410.10184/) (2024)
- [Object Attribute Matters in Visual Question Answering](https://mlsanity.com/paper/2401.09442/) (2023)
- [Overcoming Language Priors for Visual Question Answering Based on Knowledge Distillation](https://mlsanity.com/paper/2501.05690/) (2025)
- [OpenAVS: Training-Free Open-Vocabulary Audio Visual Segmentation with Foundational Models](https://mlsanity.com/paper/2505.01448/) (2025)
- [SC-ML: Self-supervised Counterfactual Metric Learning for Debiased Visual Question Answering](https://mlsanity.com/paper/2304.01647/) (2023)

## Abstract (arXiv)

A number of studies point out that current Visual Question Answering (VQA) models are severely affected by the language prior problem, which refers to blindly making predictions based on the language shortcut. Some efforts have been devoted to overcoming this issue with delicate models. However, there is no research to address it from the angle of the answer feature space learning, despite of the fact that existing VQA methods all cast VQA as a classification task. Inspired by this, in this work, we attempt to tackle the language prior problem from the viewpoint of the feature space learning. To this end, an adapted margin cosine loss is designed to discriminate the frequent and the sparse answer feature space under each question type properly. As a result, the limited patterns within the language modality are largely reduced, thereby less language priors would be introduced by our method. We apply this loss function to several baseline models and evaluate its effectiveness on two VQA-CP benchmarks. Experimental results demonstrate that our adapted margin cosine loss can greatly enhance the baseline models with an absolute performance gain of 15% on average, strongly verifying the potential of tackling the language prior problem in VQA from the angle of the answer feature space learning.