mlsanityData to 14 Oct 2026

MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations

arXiv 2608.25575 ↗First posted 26 Aug 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Pretrained vision-language models such as CLIP excel at zero-shot recognition but often fail at compositionality, particularly attribute-object and relational structures. Recent studies mitigate this issue by augmenting training with synthetic hard negatives generated by a cascade of large language models and text-to-image models, which incurs substantial pipeline overhead. We instead propose MLLMCLIP, a heterogeneous distillation framework that transfers multimodal knowledge directly from a generative Multimodal Large Language Model (MLLM) teacher into a discriminative CLIP student, bypassing synthetic data entirely. To bridge the architectural mismatch between the two paradigms, we introduce an attention-based per-layer token selection and a CKA-based distillation loss. Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 54 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

37 of 54 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining30 Apr
  2. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captions11 Oct
  3. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality26 Jun
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct
  6. 2022UniCLIP: Unified Framework for Contrastive Language-Image Pre-training27 Sep

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2013Describing Textures in the Wildcited by 2,125
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  4. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  9. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  10. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  11. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  12. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
19 more
  1. 2013Challenges in Representation Learning: A report on three machine learning contestscited by 302
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  3. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  4. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  5. 2023Improving CLIP Training with Language Rewritescited by 217
  6. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  7. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  8. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  9. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  10. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  11. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 58
  12. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 52
  13. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  14. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  15. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillationcited by 26
  16. 2024Data-Efficient Contrastive Language-Image Pretraining: Prioritizing Data Quality over Quantitycited by 19
  17. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  18. 2024ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Modelcited by 9
  19. 2023Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute associationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss
  2. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  3. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Models
  4. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  5. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
  6. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognition
  7. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
  8. 2026Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection