mlsanityData to 7 Oct 2026

COIN: Counterfactual Image Generation for VQA Interpretation

arXiv 2201.03342 ↗First posted 10 Jan 2022cs.CV cs.LG cs.MMcited by 4

Not on the map. Why?

Abstract (arXiv)

Due to the significant advancement of Natural Language Processing and Computer Vision-based models, Visual Question Answering (VQA) systems are becoming more intelligent and advanced. However, they are still error-prone when dealing with relatively complex questions. Therefore, it is important to understand the behaviour of the VQA models before adopting their results. In this paper, we introduce an interpretability approach for VQA models by generating counterfactual images. Specifically, the generated image is supposed to have the minimal possible change to the original image and leads the VQA model to give a different answer. In addition, our approach ensures that the generated image is realistic. Since quantitative metrics cannot be employed to evaluate the interpretability of the model, we carried out a user study to assess different aspects of our approach. In addition to interpreting the result of VQA models on single images, the obtained results and the discussion provides an extensive explanation of VQA models' behaviour.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 53 references that are in the map. First posted on arXiv 10 Jan 2022.

Builds on

43 of 53 references.

References · 43

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  2. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  10. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  11. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
31 more
  1. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  2. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  3. 2019Counterfactual Visual Explanationscited by 312
  4. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  5. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  6. 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 248
  7. 2018Visual Interpretability for Deep Learning: a Surveycited by 155
  8. 2020Counterfactual Samples Synthesizing for Robust Visual Question Answeringcited by 153
  9. 2017MUTAN: Multimodal Tucker Fusion for Visual Question Answeringcited by 137
  10. 2016Visual Question Answering: A Survey of Methods and Datasetscited by 123
  11. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  12. 2016Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?cited by 111
  13. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  14. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  15. 2018Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Languagecited by 90
  16. 2020Causal Interpretability for Machine Learning -- Problems, Methods and Evaluationcited by 82
  17. 2019Image Counterfactual Sensitivity Analysis for Detecting Unintended Biascited by 74
  18. 2018Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolutioncited by 66
  19. 2020Learning What Makes a Difference from Counterfactual Examples and Gradient Supervisioncited by 66
  20. 2020Overcoming Language Priors with Self-supervised Learning for Visual Question Answeringcited by 43
  21. 2018Generating Counterfactual Explanations with Natural Languagecited by 37
  22. 2020ViCE: Visual Counterfactual Explanations for Machine Learning Modelscited by 34
  23. 2018Interpretable Visual Question Answering by Visual Grounding from Attention Supervision Miningcited by 28
  24. 2020Explaining Data-Driven Decisions made by AI Systems: The Counterfactual Approachcited by 26
  25. 2018Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captionscited by 22
  26. 2020Why Spectral Normalization Stabilizes GANs: Analysis and Improvementscited by 21
  27. 2019Visual Question Answering using Deep Learning: A Survey and Performance Analysiscited by 19
  28. 2019Interpretable CNNs for Object Classificationcited by 16
  29. 2017Survey of Visual Question Answering: Datasets and Techniquescited by 9
  30. 2019Question-Conditioned Counterfactual Image Generation for VQAcited by 5
  31. 2019Image Manipulation with Natural Language using Two-sidedAttentive Conditional Generative Adversarial Networkcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1.

4 papers.

  1. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  2. 2022Rethinking Data Augmentation for Robust Visual Question Answeringcited by 18
  3. 2024Improving Data Augmentation for Robust Visual Question Answering with Effective Curriculum Learningcited by 2
  4. 2023Knowledge-Based Counterfactual Queries for Visual Question Answeringcited by 0