mlsanityData to 7 Oct 2026

CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples

arXiv 2402.13254 ↗First posted 20 Feb 2024cs.CV cs.AI cs.CLcited by 35

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We propose CounterCurate, a framework to comprehensively improve the visio-linguistic compositional reasoning capability for both contrastive and generative multimodal models. In particular, we identify two critical under-explored problems: the neglect of the physically grounded reasoning (counting and position understanding) and the potential of using highly capable text and image generation models for semantic counterfactual fine-tuning. Our work pioneers an approach that addresses these gaps. We first spotlight the near-chance performance of multimodal models like CLIP and LLaVA in physically grounded compositional reasoning. We then apply simple data augmentation using grounded image generation model GLIGEN to generate fine-tuning data, resulting in significant performance improvements: +33% and +37% for CLIP and LLaVA, respectively, on our newly curated Flickr30k-Positions benchmark. Moreover, we exploit the capabilities of high-performing text generation and image generation models, specifically GPT-4V and DALLE-3, to curate challenging semantic counterfactuals, thereby further enhancing compositional reasoning capabilities on benchmarks such as SugarCrepe, where CounterCurate outperforms GPT-4V. To facilitate future research, we release our code, dataset, benchmark, and checkpoints at https://countercurate.github.io.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 45 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

22 of 45 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2019Natural Adversarial Examplescited by 1,401
10 more
  1. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  2. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  4. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  5. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  6. 2023Teaching CLIP to Count to Tencited by 194
  7. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
  8. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  9. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  10. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 18; 2026: 8.

35 papers.

Sort:

By area:

  1. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  2. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  3. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  4. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 37
  5. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMscited by 32
  6. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  7. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  8. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  9. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Imagescited by 14
  10. 2024Deciphering the Role of Representation Disentanglement: Investigating Compositional Generalization in CLIP Modelscited by 13
  11. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
  12. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
23 more
  1. 2025Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanationscited by 12
  2. 2025DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoningcited by 10
  3. 2024Removing Distributional Discrepancies in Captions Improves Image-Text Alignmentcited by 8
  4. 2026RSGround-R1: Rethinking Remote Sensing Visual Grounding through Spatial Reasoningcited by 8
  5. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaborationcited by 7
  6. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  7. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  8. 2024Detect, Describe, Discriminate: Moving Beyond VQA for MLLM Evaluationcited by 4
  9. 2025Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Modelscited by 2
  10. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  11. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Modelscited by 2
  12. 2025SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoningcited by 2
  13. 2025ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matchingcited by 2
  14. 2025Contamination Detection for VLMs using Multi-Modal Semantic Perturbationcited by 1
  15. 2026Few Shots Text to Image Retrieval: New Benchmarking Dataset and Optimization Methodscited by 1
  16. 2026No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairscited by 1
  17. 2024Stimulating Imagination: Towards General-purpose "Something Something Placement"cited by 0
  18. 2025TinyGiantVLM: A Lightweight Vision-Language Architecture for Spatial Reasoning under Resource Constraintscited by 0
  19. 2026SynMulti: Synthetic-to-Real Learning for Multimodal Video Understandingcited by 0
  20. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0
  21. 2026MHPR: Multidimensional Human Perception and Reasoning Benchmark for Large Vision-Languate Modelscited by 0
  22. 2026RePair: Turning Retrieval Failures into Counterfactual Hard Pairscited by 0
  23. 2026EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets
  2. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
  4. 2026Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning
  5. 2024Prompting Large Vision-Language Models for Compositional Reasoning
  6. 2023What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models
  7. 2023Distilling Knowledge from Text-to-Image Generative Models Improves Visio-Linguistic Reasoning in CLIP
  8. 2026C3PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models