mlsanityData to 7 Oct 2026

COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training

arXiv 2412.01814 ↗First posted 2 Dec 2024cs.CV cs.AI cs.CLcited by 15

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) trained with contrastive loss have achieved significant advancements in various vision and language tasks. However, the global nature of the contrastive loss makes VLMs focus predominantly on foreground objects, neglecting other crucial information in the image, which limits their effectiveness in downstream tasks. To address these challenges, we propose COSMOS: CrOSs-MOdality Self-distillation for vision-language pre-training that integrates a novel text-cropping strategy and cross-attention module into a self-supervised learning framework. We create global and local views of images and texts (i.e., multi-modal augmentations), which are essential for self-distillation in VLMs. We further introduce a cross-attention module, enabling COSMOS to learn comprehensive cross-modal representations optimized via a cross-modality self-distillation loss. COSMOS consistently outperforms previous strong baselines on various zero-shot downstream tasks, including retrieval, classification, and semantic segmentation. Additionally, it surpasses CLIP-based models trained on larger datasets in visual perception and contextual understanding tasks. Code is available at https://github.com/ExplainableML/cosmos.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 76 of its 98 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

76 of 98 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024From Pixels to Prose: A Large Dataset of Dense Image Captions14 Jun
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  3. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov

Other references · 73

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  12. 2019Towards VQA Models That Can Readcited by 1,696
  13. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  14. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  16. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  17. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  18. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  19. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  20. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  21. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  22. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  23. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  24. 2021Extract Free Dense Labels from CLIPcited by 580
  25. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  26. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  27. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  28. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  29. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  30. 2023Demystifying CLIP Datacited by 265
  31. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  32. 2023Improving CLIP Training with Language Rewritescited by 216
  33. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  34. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  35. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  36. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  37. 2020Single-Stage Semantic Segmentation from Image Labelscited by 167
  38. 2022Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairscited by 130

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70
  2. 2025Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perceptioncited by 8
  3. 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMscited by 4
  4. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 2
  5. 2026Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compressioncited by 2
  6. 2025FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understandingcited by 1
  7. 2026MolSight: Molecular Property Prediction with Imagescited by 1
  8. 2025microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classificationcited by 0
  9. 2026WiFi2Cap: Semantic Action Captioning from Wi-Fi CSI via Limb-Level Semantic Alignmentcited by 0
  10. 2026GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretrainingcited by 0
  11. 2026Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Modelscited by 0
  12. 2026Multi-FRuGaL: Multimodal Flexible Redundancy-aware Decomposed Gated Learning for Cancer Diagnosis and Prognosiscited by 0
3 more
  1. 2026Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithmcited by 0
  2. 2026RPM-Distill: Physiology-guided Adaptive Cross-modal Distillation for Robust Remote Physiological Measurementcited by 0
  3. 2026COSMO: Consensus-Driven Shift Modulation for Source-Free Domain Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion
  2. 2026LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
  3. 2025Cross-Modal Attention Guided Unlearning in Vision-Language Models
  4. 2025Efficiently Integrate Large Language Models with Visual Perception: A Survey from the Training Paradigm Perspective
  5. 2026Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
  6. 2026LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
  7. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
  8. 2024CLIP-CID: Efficient CLIP Distillation via Cluster-Instance Discrimination