mlsanityData to 7 Oct 2026

Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models

arXiv 2305.19595 ↗First posted 31 May 2023cs.CVcited by 72

Not on the map. Why?

Cited by 33 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision and Language (VL) models offer an effective method for aligning representation spaces of images and text, leading to numerous applications such as cross-modal retrieval, visual question answering, captioning, and more. However, the aligned image-text spaces learned by all the popular VL models are still suffering from the so-called `object bias' - their representations behave as `bags of nouns', mostly ignoring or downsizing the attributes, relations, and states of objects described/appearing in texts/images. Although some great attempts at fixing these `compositional reasoning' issues were proposed in the recent literature, the problem is still far from being solved. In this paper, we uncover two factors limiting the VL models' compositional reasoning performance. These two factors are properties of the paired VL dataset used for finetuning and pre-training the VL model: (i) the caption quality, or in other words `image-alignment', of the texts; and (ii) the `density' of the captions in the sense of mentioning all the details appearing on the image. We propose a fine-tuning approach for automatically treating these factors leveraging a standard VL dataset (CC3M). Applied to CLIP, we demonstrate its significant compositional reasoning performance increase of up to ~27% over the base model, up to ~20% over the strongest baseline, and by 6.7% on average.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 69 of its 87 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

69 of 87 references.

References · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2015Learning Deep Features for Discriminative Localizationcited by 2,430
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  7. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  12. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  13. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  14. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  15. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  16. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  17. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  18. 2021Image Segmentation Using Text and Image Promptscited by 509
  19. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  20. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  21. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  22. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  23. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  24. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  25. 2018Image Generation from Scene Graphscited by 324
  26. 2022Visual Spatial Reasoningcited by 288
  27. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  28. 2018Videos as Space-Time Region Graphscited by 221
  29. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  30. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  31. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  32. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  33. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  34. 2019Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networkscited by 112
  35. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretrainingcited by 104
  36. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  37. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  38. 2020Grounded Situation Recognitioncited by 77

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 29; 2025: 23; 2026: 12.

72 papers.

Sort:

By area:

  1. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  2. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  3. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  4. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  5. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  6. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  7. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 40
  8. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  9. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
  10. 2023Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Miningcited by 25
  11. 2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 24
  12. 2024VisMin: Visual Minimal-Change Understandingcited by 23
38 more
  1. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  2. 2025Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferencescited by 22
  3. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  4. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMscited by 21
  5. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 21
  6. 2023CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decodingcited by 20
  7. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  8. 2025UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarkscited by 17
  9. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
  10. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  11. 2024Hyperbolic Learning with Synthetic Captions for Open-World Detectioncited by 11
  12. 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Modelscited by 9
  13. 2024Mosaic-IT: Cost-Free Compositional Data Synthesis for Instruction Tuningcited by 9
  14. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  15. 2024Unified Lexical Representation for Interpretable Visual-Language Alignmentcited by 8
  16. 2024Removing Distributional Discrepancies in Captions Improves Image-Text Alignmentcited by 8
  17. 2025TTRV: Test-Time Reinforcement Learning for Vision Language Modelscited by 8
  18. 2024Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captionscited by 7
  19. 2024GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Modelscited by 6
  20. 2024Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videoscited by 6
  21. 2024Teaching VLMs to Localize Specific Objects from In-context Examplescited by 6
  22. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  23. 2024No Detail Left Behind: Revisiting Self-Retrieval for Fine-Grained Image Captioningcited by 5
  24. 2024Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Modelscited by 5
  25. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  26. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  27. 2025Weakly Supervised Video Scene Graph Generation via Natural Language Supervisioncited by 4
  28. 2025Common Data Properties Limit Object-Attribute Binding in CLIPcited by 4
  29. 2025TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Modelscited by 4
  30. 2025VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenescited by 4
  31. 2024Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Modelscited by 3
  32. 2024SHINE: Saliency-aware HIerarchical NEgative Ranking for Compositional Temporal Groundingcited by 3
  33. 2024Weak-to-Strong Compositional Learning from Generative Models for Language-based Object Detectioncited by 3
  34. 2024Detailed Object Description with Controllable Dimensionscited by 3
  35. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  36. 2024Exploring the Spectrum of Visio-Linguistic Compositionality and Recognitioncited by 2
  37. 2025SCRAMBLe : Enhancing Multimodal LLM Compositionality with Synthetic Preference Datacited by 2
  38. 2025Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Modelscited by 2

50 most cited of 72 All 72 (JSON)