mlsanityData to 7 Oct 2026

In-Context Learning Improves Compositional Understanding of Vision-Language Models

arXiv 2407.15487 ↗First posted 22 Jul 2024cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-Language Models (VLMs) have shown remarkable capabilities in a large number of downstream tasks. Nonetheless, compositional image understanding remains a rather difficult task due to the object bias present in training data. In this work, we investigate the reasons for such a lack of capability by performing an extensive bench-marking of compositional understanding in VLMs. We compare contrastive models with generative ones and analyze their differences in architecture, pre-training data, and training tasks and losses. Furthermore, we leverage In-Context Learning (ICL) as a way to improve the ability of VLMs to perform more complex reasoning and understanding given an image. Our extensive experiments demonstrate that our proposed approach outperforms baseline models across multiple compositional understanding datasets.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 29 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

27 of 29 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality26 Jun
  2. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
13 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  2. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  3. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  6. 2021MetaICL: Learning to Learn In Contextcited by 450
  7. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  8. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  9. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  10. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  11. 2022Patch-level Representation Learning for Self-supervised Vision Transformerscited by 60
  12. 2023Neurosymbolic Grounding for Compositional World Modelscited by 14
  13. 2024PIN: Positional Insert Unlocks Object Localisation Abilities in VLMscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026DreamActor-M2: Universal Character Image Animation via Spatiotemporal In-Context Learningcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding
  2. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks
  3. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysis
  4. 2025COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Models
  5. 20233VL: Using Trees to Improve Vision-Language Models' Interpretability
  6. 2025Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability
  7. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models
  8. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding