mlsanityData to 7 Oct 2026

The Role of Linguistic Priors in Measuring Compositional Generalization of Vision-Language Models

arXiv 2310.02777 ↗First posted 4 Oct 2023cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Compositionality is a common property in many modalities including natural languages and images, but the compositional generalization of multi-modal models is not well-understood. In this paper, we identify two sources of visual-linguistic compositionality: linguistic priors and the interplay between images and texts. We show that current attempts to improve compositional generalization rely on linguistic priors rather than on information in the image. We also propose a new metric for compositionality without such linguistic priors.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 34 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

26 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  10. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  11. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  12. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
13 more
  1. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  2. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  3. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  4. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  5. 2019Measuring Compositionality in Representation Learningcited by 84
  6. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  7. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34
  8. 2021Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQAcited by 30
  9. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25
  10. 2022When classifying grammatical role, BERT doesn't care about word order... except when it matterscited by 24
  11. 2022Measuring and Improving Compositional Generalization in Text-to-SQL via Component Alignmentcited by 16
  12. 2018Defoiling Foiled Image Captionscited by 9
  13. 2022Word Order Matters when you Increase Maskingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  2. 2026ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrievalcited by 3
  3. 2026TangramSR: Can Vision-Language Models Reason in Continuous Geometric Space?cited by 1
  4. 2026Don't CLAP: Are Music-Text Models Bag-of-Words?cited by 0
  5. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?
  2. 2025Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models
  3. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  4. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models
  5. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Models
  6. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
  7. 2024Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition
  8. 2024ComAlign: Compositional Alignment in Vision-Language Models