# SC-ML: Self-supervised Counterfactual Metric Learning for Debiased Visual Question Answering

arXiv: https://arxiv.org/abs/2304.01647 · HTML: https://mlsanity.com/paper/2304.01647/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 4 Apr 2023.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [RUBi: Reducing Unimodal Biases in Visual Question Answering](https://mlsanity.com/paper/1906.10169/) (2019)
- [Counterfactual Samples Synthesizing for Robust Visual Question Answering](https://mlsanity.com/paper/2003.06576/) (2020)
- [Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification](https://mlsanity.com/paper/2108.08728/) (2021)
- [Overcoming Language Priors in Visual Question Answering with Adversarial Regularization](https://mlsanity.com/paper/1810.03649/) (2018)
- [Ranked List Loss for Deep Metric Learning](https://mlsanity.com/paper/1903.03238/) (2019)
- [MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering](https://mlsanity.com/paper/2009.08566/) (2020)
- [Overcoming Language Priors with Self-supervised Learning for Visual Question Answering](https://mlsanity.com/paper/2012.11528/) (2020)
- [Counterfactual Samples Synthesizing and Training for Robust Visual Question Answering](https://mlsanity.com/paper/2110.01013/) (2021)
- [AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss](https://mlsanity.com/paper/2105.01993/) (2021)
- [Check It Again: Progressive Visual Question Answering via Visual Entailment](https://mlsanity.com/paper/2106.04605/) (2021)

## Abstract (arXiv)

Visual question answering (VQA) is a critical multimodal task in which an agent must answer questions according to the visual cue. Unfortunately, language bias is a common problem in VQA, which refers to the model generating answers only by associating with the questions while ignoring the visual content, resulting in biased results. We tackle the language bias problem by proposing a self-supervised counterfactual metric learning (SC-ML) method to focus the image features better. SC-ML can adaptively select the question-relevant visual features to answer the question, reducing the negative influence of question-irrelevant visual features on inferring answers. In addition, question-irrelevant visual features can be seamlessly incorporated into counterfactual training schemes to further boost robustness. Extensive experiments have proved the effectiveness of our method with improved results on the VQA-CP dataset. Our code will be made publicly available.