mlsanityData to 14 Oct 2026

When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA

arXiv 2511.17886 ↗First posted 22 Nov 2025cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) have achieved remarkable success across multimodal tasks, yet their substantial computational demands hinder efficient deployment. Knowledge distillation (KD) has emerged as a powerful approach for building lightweight but competitive models, with strong evidence from both language and vision domains. However, its application to VLMs, particularly CLIP-style models, remains limited, often constrained to small-scale teachers and narrow evaluation tasks such as classification or retrieval. In this work, we present the first systematic study of distillation across a range of CLIP-style teacher models, ranging from standard baselines to large-scale state-of-the-art models. Contrary to trends observed in NLP and vision, we find that stronger teachers do not consistently yield better students; in fact, existing distillation frameworks often fail to scale, leading to degraded performance in downstream multimodal tasks such as visual question answering. Our findings challenge prevailing assumptions in KD and point toward new directions for designing parameter-efficient multimodal models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 48 references that are in the map. First posted on arXiv 22 Nov 2025.

Builds on

41 of 48 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Models21 Oct
  2. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning30 Sep
  3. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation28 Aug
  4. 2024LLAVADI: What Matters For Multimodal Large Language Models Distillation28 Jul
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  6. 2024Efficient Multimodal Large Language Models: A Survey17 May
  7. 2024Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models10 Mar
  8. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  9. 2024Efficient Multimodal Learning from Data-centric Perspective18 Feb
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  9. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  10. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  12. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
18 more
  1. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  2. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  3. 2023Faster Segment Anything: Towards Lightweight SAM for Mobile Applicationscited by 352
  4. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  5. 2021SEED: Self-supervised Distillation For Visual Representationcited by 134
  6. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 129
  7. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  8. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  9. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  10. 2021Similarity and Matching of Neural Network Representationscited by 78
  11. 2025Distillation Scaling Lawscited by 67
  12. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  13. 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
  14. 2025A Survey on Efficient Vision-Language Modelscited by 19
  15. 2022Curriculum Learning for Data-Efficient Vision-Language Alignmentcited by 11
  16. 2024Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Biascited by 7
  17. 2022Feature Structure Distillation with Centered Kernel Alignment in BERT Transferringcited by 6
  18. 2024Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Trainingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
  2. 2025Enriching Knowledge Distillation with Cross-Modal Teacher Fusion
  3. 2026Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement
  4. 2026Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
  5. 2024CLIP-Embed-KD: Computationally Efficient Knowledge Distillation Using Embeddings as Teachers
  6. 2026The Professor: Multi-Teacher Unsupervised Prompt Distillation for Vision-Language Models
  7. 2025AMMKD: Adaptive Multimodal Multi-teacher Distillation for Lightweight Vision-Language Models
  8. 2023Can a student Large Language Model perform as well as it's teacher?