mlsanityData to 7 Oct 2026

LLAVADI: What Matters For Multimodal Large Language Models Distillation

arXiv 2407.19409 ↗First posted 28 Jul 2024cs.CL cs.CVcited by 25

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The recent surge in Multimodal Large Language Models (MLLMs) has showcased their remarkable potential for achieving generalized intelligence by integrating visual understanding into Large Language Models.Nevertheless, the sheer model size of MLLMs leads to substantial memory and computational demands that hinder their widespread deployment. In this work, we do not propose a new efficient model structure or train small-scale MLLMs from scratch. Instead, we focus on what matters for training small-scale MLLMs through knowledge distillation, which is the first step from the multimodal distillation perspective. Our extensive studies involve training strategies, model choices, and distillation algorithms in the knowledge distillation process. These results show that joint alignment for both tokens and logit alignment plays critical roles in teacher-student frameworks. In addition, we draw a series of intriguing observations from this study. By evaluating different benchmarks and proper strategy, even a 2.7B small-scale model can perform on par with larger models with 7B or 13B parameters. Our code and models will be publicly available for further research.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 63 of its 76 references that are in the map. First posted on arXiv 28 Jul 2024.

Builds on

63 of 76 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  2. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones28 Dec
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023VIGC: Visual Instruction Generation and Correction24 Aug
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 54

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2017Decoupled Weight Decay Regularizationcited by 1,722
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  8. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  12. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  13. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  15. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  16. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  17. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  18. 2016Visual Dialogcited by 377
  19. 2022Visual Spatial Reasoningcited by 288
  20. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  21. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  22. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  23. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 188
  24. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 161
  25. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  26. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  27. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  28. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  29. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  30. 2024Logit Standardization in Knowledge Distillationcited by 101
  31. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  32. 2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
  33. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
  34. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  35. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  36. 2018Improving Fast Segmentation With Teacher-student Learningcited by 34
  37. 2023MFTCoder: Boosting Code LLMs with Multitask Fine-Tuningcited by 34
  38. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Taskscited by 32

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 10.

25 papers.

Sort:

By area:

  1. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  2. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  3. 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMscited by 13
  4. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  5. 2025SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignmentcited by 9
  6. 2025MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modelingcited by 6
  7. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  8. 2026Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactionscited by 5
  9. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  10. 2025EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokenscited by 4
  11. 2025GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Modelscited by 2
  12. 2025Towards Long-window Anchoring in Vision-Language Model Distillationcited by 2
13 more
  1. 2025ReGATE: Learning Faster and Better with Fewer Tokens in MLLMscited by 1
  2. 2026Robustness of Vision Language Models Against Split-Image Harmful Input Attackscited by 1
  3. 2026OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transportcited by 1
  4. 2026Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Modelscited by 1
  5. 2026Deep Thought Alignment: Trajectory-Level Latent Distillation for Video Reasoningcited by 1
  6. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMscited by 0
  7. 2025Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Languagecited by 0
  8. 2025When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQAcited by 0
  9. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Modelscited by 0
  10. 2026Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillationcited by 0
  11. 2026C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMscited by 0
  12. 2026MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representationscited by 0
  13. 2026MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation
  2. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  3. 2024The Revolution of Multimodal Large Language Models: A Survey
  4. 2026KDFP: A first-principles approach to knowledge distillation in large language models
  5. 2024AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability
  6. 2024Model Composition for Multimodal Large Language Models
  7. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data
  8. 2025SAISA: Towards Multimodal Large Language Models with Both Training and Inference Efficiency