# Counterfactual Reasoning for Robust Visual Question Answering

arXiv: https://arxiv.org/abs/2609.16567 · HTML: https://mlsanity.com/paper/2609.16567/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 15 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases](https://mlsanity.com/paper/1909.03683/) (2019)
- [Counterfactual Samples Synthesizing for Robust Visual Question Answering](https://mlsanity.com/paper/2003.06576/) (2020)
- [Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded](https://mlsanity.com/paper/1902.03751/) (2019)
- [Analyzing the Behavior of Visual Question Answering Models](https://mlsanity.com/paper/1606.07356/) (2016)
- [Overcoming Language Priors in Visual Question Answering with Adversarial Regularization](https://mlsanity.com/paper/1810.03649/) (2018)
- [MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering](https://mlsanity.com/paper/2009.08566/) (2020)
- [Self-Critical Reasoning for Robust Visual Question Answering](https://mlsanity.com/paper/1905.09998/) (2019)
- [Greedy Gradient Ensemble for Robust Visual Question Answering](https://mlsanity.com/paper/2107.12651/) (2021)
- [Rethinking Data Augmentation for Robust Visual Question Answering](https://mlsanity.com/paper/2207.08739/) (2022)
- [Generative Bias for Robust Visual Question Answering](https://mlsanity.com/paper/2208.00690/) (2022)
- [Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning](https://mlsanity.com/paper/2210.04563/) (2022)
- [Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention](https://mlsanity.com/paper/2410.10184/) (2024)

## Abstract (arXiv)

Modern Visual Question Answering (VQA) models often exploit spurious correlations in training data, leading to poor out-of-distribution (OOD) generalization due to language bias. Although counterfactual learning has shown promise, existing methods can be improved to better guide attention toward causal evidence and strengthen feature discrimination. To address this, we propose a novel training framework that enhances counterfactual contrastive learning for VQA. Our framework introduces three key contributions: (1) a three-stage curriculum for stable multi-objective optimization, (2) an enhanced Batch-Contrastive loss for more discriminative feature learning, and (3) two novel regularizers, Answer-Contrastive (AC) loss to refine the prediction space and Gradient-Discrepancy (GD) loss to enforce causal visual grounding. Our model achieves a competitive accuracy of 61.64% on the bias-sensitive VQA-CP v2 benchmark while maintaining 62.80% on the standard VQA v2 dataset, yielding a small generalization gap of 1.16%. This demonstrates a strong balance between OOD robustness and in-distribution performance.