mlsanityData to 7 Oct 2026

When and why vision-language models behave like bags-of-words, and what to do about it?

arXiv 2210.01936 ↗First posted 4 Oct 2022cs.CV cs.AI cs.CLcited by 603

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode compositional information. Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order. ARO consists of Visual Genome Attribution, to test the understanding of objects' properties; Visual Genome Relation, to test for relational understanding; and COCO & Flickr30k-Order, to test for order sensitivity. ARO is orders of magnitude larger than previous benchmarks of compositionality, with more than 50,000 test cases. We show where state-of-the-art VLMs have poor relational understanding, can blunder when linking objects to their attributes, and demonstrate a severe lack of order sensitivity. VLMs are predominantly trained and evaluated on large datasets with rich compositional structure in the images and captions. Yet, training on these datasets has not been enough to address the lack of compositional understanding, and evaluating on these datasets has failed to surface this deficiency. To understand why these limitations emerge and are not represented in the standard tests, we zoom into the evaluation and training procedures. We demonstrate that it is possible to perform well on retrieval over existing datasets without using the composition and order information. Given that contrastive pretraining optimizes for retrieval on datasets with similar shortcuts, we hypothesize that this can explain why the models do not need to learn to represent compositional information. This finding suggests a natural solution: composition-aware hard negative mining. We show that a simple-to-implement modification of contrastive learning significantly improves the performance on tasks requiring understanding of order and compositionality.

Key facts. Cited by 603 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 63 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

45 of 63 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
33 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  3. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  6. 2020Contrastive Learning with Hard Negative Samplescited by 619
  7. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  9. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  10. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  11. 2020Hard Negative Mixing for Contrastive Learningcited by 399
  12. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  13. 2017Sampling Matters in Deep Embedding Learningcited by 285
  14. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  15. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  16. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  17. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  18. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  19. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  20. 2018Deep Metric Learning with Hierarchical Triplet Losscited by 112
  21. 2017Smart Mining for Deep Metric Learningcited by 96
  22. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  23. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  24. 2022Testing Relational Understanding in Text-Guided Image Generationcited by 59
  25. 2021What Context Features Can Transformer Language Models Use?cited by 49
  26. 2021Mitigating Dataset Harms Requires Stewardship: Lessons from 1000 Paperscited by 49
  27. 2022A Benchmark for Compositional Visual Reasoningcited by 43
  28. 2021Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentationcited by 40
  29. 2022American == White in Multimodal Language-and-Image AIcited by 39
  30. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34
  31. 2021About Face: A Survey of Facial Recognition Evaluationcited by 27
  32. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25
  33. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 86; 2024: 179; 2025: 173; 2026: 164.

603 papers.

Sort:

By area:

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  3. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  4. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  5. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  6. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  7. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  8. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  9. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  10. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  11. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  12. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
38 more
  1. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
  2. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  3. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignmentcited by 169
  4. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  5. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 134
  6. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  7. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  8. 2023SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensingcited by 123
  9. 2023Aligning Bag of Regions for Open-Vocabulary Object Detectioncited by 119
  10. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  11. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  12. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  13. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  14. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  15. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  16. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  17. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  18. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 94
  19. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  20. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
  21. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  22. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  23. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  24. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  25. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  26. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  27. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  28. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  29. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  30. 2024Compositional Entailment Learning for Hyperbolic Vision-Language Modelscited by 72
  31. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70
  32. 2025Vision-Language Models Do Not Understand Negationcited by 69
  33. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  34. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  35. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  36. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  37. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  38. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56

50 most cited of 603 All 603 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  2. 2024RelationVLM: Making Large Vision-Language Models Understand Visual Relations
  3. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability
  4. 2025Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability
  5. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Models
  6. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models
  7. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?
  8. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Models