mlsanityData to 7 Oct 2026

COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs

arXiv 2309.14356 ↗First posted 23 Sep 2023cs.LG cs.CL cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

Counterfactual examples have proven to be valuable in the field of natural language processing (NLP) for both evaluating and improving the robustness of language models to spurious correlations in datasets. Despite their demonstrated utility for NLP, multimodal counterfactual examples have been relatively unexplored due to the difficulty of creating paired image-text data with minimal counterfactual changes. To address this challenge, we introduce a scalable framework for automatic generation of counterfactual examples using text-to-image diffusion models. We use our framework to create COCO-Counterfactuals, a multimodal counterfactual dataset of paired image and text captions based on the MS-COCO dataset. We validate the quality of COCO-Counterfactuals through human evaluations and show that existing multimodal models are challenged by our counterfactual image-text pairs. Additionally, we demonstrate the usefulness of COCO-Counterfactuals for improving out-of-domain generalization of multimodal vision-language models via training data augmentation.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 41 of its 52 references that are in the map. First posted on arXiv 23 Sep 2023.

Builds on

41 of 52 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
29 more
  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  2. 2019Invariant Risk Minimizationcited by 1,832
  3. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  6. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  7. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  9. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  10. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  11. 2020Noise or Signal: The Role of Image Backgrounds in Object Recognitioncited by 342
  12. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  13. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  14. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  15. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  16. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 225
  17. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  18. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  19. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  20. 2021Salient ImageNet: How to discover spurious features in Deep Learning?cited by 114
  21. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  22. 2020Robustness to Spurious Correlations in Text Classification via Automatically Generated Counterfactualscited by 67
  23. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  24. 2021An Investigation of the (In)effectiveness of Counterfactually Augmented Datacited by 42
  25. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  26. 2022DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptationcited by 36
  27. 2023Dataset Interfaces: Diagnosing Model Failures Using Controllable Counterfactual Generationcited by 35
  28. 2021Exploring the Efficacy of Automatically Generated Counterfactuals for Sentiment Analysiscited by 30
  29. 2022NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentationcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 17; 2025: 12; 2026: 7.

39 papers.

Sort:

By area:

  1. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  2. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  3. 2024MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMscited by 60
  4. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  5. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  6. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 37
  7. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  8. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 30
  9. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  10. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  11. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  12. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
27 more
  1. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  2. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  3. 2024Social Debiasing for Fair Multi-modal LLMscited by 8
  4. 2024Removing Distributional Discrepancies in Captions Improves Image-Text Alignmentcited by 8
  5. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaborationcited by 7
  6. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  7. 2023Probing Intersectional Biases in Vision-Language Models with Counterfactual Examplescited by 5
  8. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Imagescited by 5
  9. 2025On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Studycited by 5
  10. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  11. 2025Causally Steered Diffusion for Automated Video Counterfactual Generationcited by 5
  12. 2025Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routingcited by 5
  13. 2026Can Large Language Models Still Explain Themselves? Investigating the Impact of Quantization on Self-Explanationscited by 5
  14. 2024CounterFace: A Synthetic Face Dataset for Fine-Grained Counterfactual Evaluation of Face Recognition Systemscited by 2
  15. 2024Few-Class Arena: A Benchmark for Efficient Selection of Vision Models and Dataset Difficulty Measurementcited by 2
  16. 2025Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQAcited by 2
  17. 2026Seeing vs. Believing: Evaluating the Language Bias of Open-Source MLLMs in Counter-Intuitive Scenescited by 2
  18. 2026Toward Autonomous Laboratory Safety Monitoring with Vision Language Models: Learning to See Hazards Through Scene Structurecited by 2
  19. 2024Refusal as Silence: Gendered Disparities in Vision-Language Model Responsescited by 1
  20. 2026Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generationcited by 1
  21. 2024See or Guess: Counterfactually Regularized Image Captioningcited by 0
  22. 2024Causal Fine-Tuning under Latent Confounded Shiftcited by 0
  23. 2025Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generationcited by 0
  24. 2025Cross-modal Counterfactual Explanations: Uncovering Decision Factors and Dataset Biases in Subjective Classificationcited by 0
  25. 2026Half-Truths Break Similarity-Based Retrievalcited by 0
  26. 2026Concept-based Visual Counterfactual Explanations with Diffusion Modelscited by 0
  27. 2026RePair: Turning Retrieval Failures into Counterfactual Hard Pairscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Text-to-Image Models for Counterfactual Explanations: a Black-Box Approach
  2. 2026What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining
  3. 2026Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning
  4. 2025A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets
  5. 2025Automated Prompt Generation for Creative and Counterfactual Text-to-image Synthesis
  6. 2022Diffusion Models for Counterfactual Explanations
  7. 2025V-CECE: Visual Counterfactual Explanations via Conceptual Edits
  8. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals