mlsanityData to 7 Oct 2026

Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing

arXiv 1912.07538 ↗First posted 16 Dec 2019cs.CV cs.CL cs.LGcited by 108

Not on the map. Why?

Abstract (arXiv)

Despite significant success in Visual Question Answering (VQA), VQA models have been shown to be notoriously brittle to linguistic variations in the questions. Due to deficiencies in models and datasets, today's models often rely on correlations rather than predictions that are causal w.r.t. data. In this paper, we propose a novel way to analyze and measure the robustness of the state of the art models w.r.t semantic visual variations as well as propose ways to make models more robust against spurious correlations. Our method performs automated semantic image manipulations and tests for consistency in model predictions to quantify the model robustness as well as generate synthetic data to counter these problems. We perform our analysis on three diverse, state of the art VQA models and diverse question types with a particular focus on challenging counting questions. In addition, we show that models can be made significantly more robust against inconsistent predictions using our edited data. Finally, we show that results also translate to real-world error cases of state of the art models, which results in improved overall performance.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 20; 2023: 18; 2024: 16; 2025: 14; 2026: 13.

108 papers.

Sort:

By area:

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  2. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  3. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  4. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  5. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  6. 2021Interventional Video Grounding with Dual Contrastive Learningcited by 94
  7. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  8. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  9. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 74
  10. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  11. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  12. 2021BiaSwap: Removing dataset bias with bias-tailored swapping augmentationcited by 70
38 more
  1. 2021Editing a classifier by rewriting its prediction rulescited by 64
  2. 2021Human-Adversarial Visual Question Answeringcited by 57
  3. 2022Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answeringcited by 57
  4. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  5. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 55
  6. 2021Directional Bias Amplificationcited by 54
  7. 2021Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answeringcited by 50
  8. 2022Counterfactual Cycle-Consistent Learning for Instruction Following and Generation in Vision-Language Navigationcited by 50
  9. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
  10. 2024The Clever Hans Mirage: A Comprehensive Survey on Spurious Correlations in Machine Learningcited by 39
  11. 2021Finding and Fixing Spurious Patterns with Explanationscited by 38
  12. 2023Exposing and Mitigating Spurious Correlations for Cross-Modal Retrievalcited by 35
  13. 2022Invariant Feature Learning for Generalized Long-Tailed Classificationcited by 34
  14. 2023Contrastive Video Question Answering via Video Graph Transformercited by 32
  15. 2021Towards Robust Classification Model by Counterfactual and Invariant Data Generationcited by 30
  16. 2025AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decodingcited by 26
  17. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  18. 2022Causal Reasoning Meets Visual Representation Learning: A Prospective Studycited by 24
  19. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  20. 2021Counterfactual Samples Synthesizing and Training for Robust Visual Question Answeringcited by 22
  21. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  22. 2022Men Also Do Laundry: Multi-Attribute Bias Amplificationcited by 21
  23. 2021Data Efficient Masked Language Modeling for Vision and Languagecited by 18
  24. 2022Rethinking Data Augmentation for Robust Visual Question Answeringcited by 18
  25. 2022CPL: Counterfactual Prompt Learning for Vision and Language Modelscited by 18
  26. 2023Causal Unsupervised Semantic Segmentationcited by 18
  27. 2023Visual Causal Scene Refinement for Video Question Answeringcited by 16
  28. 2021Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventionscited by 15
  29. 2021Are Bias Mitigation Techniques for Deep Learning Effective?cited by 14
  30. 2021Discovering the Unknown Knowns: Turning Implicit Knowledge in the Dataset into Explicit Training Examples for Visual Question Answeringcited by 14
  31. 2022CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answeringcited by 13
  32. 2024Causal Mode Multiplexer: A Novel Framework for Unbiased Multispectral Pedestrian Detectioncited by 13
  33. 2022TAG: Boosting Text-VQA via Text-aware Visual Question-answer Generationcited by 12
  34. 2023Causal reasoning in typical computer vision taskscited by 12
  35. 2024Fairness and Bias Mitigation in Computer Vision: A Surveycited by 12
  36. 2024What makes a good metric? Evaluating automatic metrics for text-to-image consistencycited by 11
  37. 2021Detecting Spurious Correlations with Sanity Tests for Artificial Intelligence Guided Radiology Systemscited by 10
  38. 2022A Survey of Deep Causal Models and Their Industrial Applicationscited by 10

50 most cited of 108 All 108 (JSON)