mlsanityData to 13 Oct 2026

Data or Language Supervision: What Makes CLIP Better than DINO?

arXiv 2510.11835 ↗First posted 13 Oct 2025cs.CV cs.AI cs.CLcited by 13

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs), but it remains unclear whether this advantage stems from CLIP's language supervision or its much larger training data. To disentangle these factors, we pre-train CLIP and DINO under controlled settings -- using the same architecture, dataset, and training configuration -- achieving similar ImageNet accuracy. Embedding analysis shows that CLIP captures high-level semantics (e.g., object categories, text), while DINO is more responsive to low-level features like colors and styles. When integrated into VLMs and evaluated on 20 VQA benchmarks, CLIP excels at text-intensive tasks, while DINO slightly outperforms on vision-centric ones. Variants of language supervision (e.g., sigmoid loss, pre-trained language encoders) yield limited gains. Our findings provide scientific insights into vision encoder design and its impact on VLM performance.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 23 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

13 of 23 references.

References · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  6. 2019Towards VQA Models That Can Readcited by 1,705
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 827
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 792
  9. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 337
  10. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  11. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  12. 2025Scaling Language-Free Visual Representation Learningcited by 63
1 more
  1. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 28

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Linguistics and Human Brain: A Perspective of Computational Neurosciencecited by 4
  2. 2026Revisiting Multi-Task Visual Representation Learningcited by 2
  3. 2026Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Taskscited by 2
  4. 2026MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Imagecited by 2
  5. 2026CLIMP: Contrastive Language-Image Mamba Pretrainingcited by 1
  6. 2026CoME-VL: Scaling Complementary Multi-Encoder Vision-Language Learningcited by 1
  7. 2026Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurementcited by 0
  8. 2026Learning through Creation: A Hash-Free Framework for On-the-Fly Category Discoverycited by 0
  9. 2026DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classificationcited by 0
  10. 2026Are Natural-Domain Foundation Models Effective for Accelerated Cardiac MRI Reconstruction?cited by 0
  11. 2026Noise-Aware Framework for Correcting Corrupted Labelscited by 0
  12. 2026UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Modelscited by 0
1 more
  1. 2026ProGuT: Label-Efficient Panoptic Segmentation for Forest Scenescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models
  2. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
  3. 2023CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement
  4. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  5. 2026VL-DINO: Leveraging CLIP Vision-Language Knowledge for Open-Vocabulary Object Detectio
  6. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study
  7. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
  8. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias