mlsanityData to 14 Oct 2026

Counterfactual VQA: A Cause-Effect Look at Language Bias

arXiv 2006.04315 ↗First posted 8 Jun 2020cs.CV cs.CLcited by 266

Not on the map. Why?

Cited by 49 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods proposed to exclude the language prior during inference. However, they fail to disentangle the "good" language context and "bad" language bias from the whole. In this paper, we investigate how to mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers and reduce the language bias by subtracting the direct language effect from the total causal effect. Experiments demonstrate that our proposed counterfactual inference framework 1) is general to various VQA backbones and fusion strategies, 2) achieves competitive performance on the language-bias sensitive VQA-CP dataset while performs robustly on the balanced VQA v2 dataset without any augmented data. The code is available at https://github.com/yuleiniu/cfvqa.

Key facts. Cited by 266 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 34; 2022: 55; 2023: 51; 2024: 43; 2025: 43; 2026: 40.

266 papers.

Sort:

By area:

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  2. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  3. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  4. 2021Distilling Causal Effect of Data in Class-Incremental Learningcited by 117
  5. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  6. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  7. 2021Causal Attention for Vision-Language Taskscited by 111
  8. 2021Causal Attention for Interpretable and Generalizable Graph Classificationcited by 108
  9. 2021Interpretable Image Classification with Differentiable Prototypes Assignmentcited by 97
  10. 2021Counterfactual Zero-Shot and Open-Set Visual Recognitioncited by 95
  11. 2021Interventional Video Grounding with Dual Contrastive Learningcited by 94
  12. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 92
38 more
  1. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
  2. 2021Causal Attention for Unbiased Visual Recognitioncited by 80
  3. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  4. 2022Learning and Evaluating Graph Neural Network Explanations based on Counterfactual and Factual Reasoningcited by 79
  5. 2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
  6. 2022Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answeringcited by 57
  7. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 52
  8. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  9. 2024Vision-and-Language Navigation via Causal Learningcited by 51
  10. 2022Grounding Answers for Visual Questions Asked by Visually Impaired Peoplecited by 47
  11. 2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
  12. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  13. 2024VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generationcited by 41
  14. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
  15. 2025MLLMs are Deeply Affected by Modality Biascited by 40
  16. 2022Gender and Racial Bias in Visual Question Answering Datasetscited by 39
  17. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  18. 2022SwapMix: Diagnosing and Regularizing the Over-Reliance on Visual Context in Visual Question Answeringcited by 38
  19. 2023Backdoor Defense via Deconfounded Representation Learningcited by 38
  20. 2021Factual Consistency Evaluation for Text Summarization via Counterfactual Estimationcited by 37
  21. 2023Prompting Large Language Models for Counterfactual Generation: An Empirical Studycited by 37
  22. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 36
  23. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  24. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  25. 2023ProtoSeg: Interpretable Semantic Segmentation with Prototypical Partscited by 33
  26. 2023Contrastive Video Question Answering via Video Graph Transformercited by 32
  27. 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learningcited by 31
  28. 2022Revisiting Open World Object Detectioncited by 30
  29. 2022Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQAcited by 30
  30. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 30
  31. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 30
  32. 2026When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAscited by 30
  33. 2022Counterfactual harmcited by 29
  34. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  35. 2025Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcutscited by 29
  36. 2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoningcited by 28
  37. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  38. 2025Mitigating Object Hallucinations via Sentence-Level Early Interventioncited by 28

50 most cited of 266 All 266 (JSON)