# LLAVADI: What Matters For Multimodal Large Language Models Distillation

arXiv: https://arxiv.org/abs/2407.19409 · HTML: https://mlsanity.com/paper/2407.19409/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 63 of its 76 references that are in the map.
- First posted on arXiv 28 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 63)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models](https://mlsanity.com/paper/2403.06764/) (2024)

## Built on by (most cited first, 26 of 26)

- [DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models](https://mlsanity.com/paper/2503.02175/) (2025)
- [LLaVA-KD: A Framework of Distilling Multimodal Large Language Models](https://mlsanity.com/paper/2410.16236/) (2024)
- [CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2510.12184/) (2025)
- [VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models](https://mlsanity.com/paper/2412.01822/) (2024)
- [SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment](https://mlsanity.com/paper/2509.03934/) (2025)
- [MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modeling](https://mlsanity.com/paper/2503.13440/) (2025)
- [Masking Teacher and Reinforcing Student for Distilling Vision-Language Models](https://mlsanity.com/paper/2512.22238/) (2025)
- [EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens](https://mlsanity.com/paper/2511.21106/) (2025)
- [Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions](https://mlsanity.com/paper/2602.09483/) (2026)
- [Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks](https://mlsanity.com/paper/2504.17421/) (2025)
- [OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport](https://mlsanity.com/paper/2602.20205/) (2026)
- [GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models](https://mlsanity.com/paper/2506.13166/) (2025)
- [Towards Long-window Anchoring in Vision-Language Model Distillation](https://mlsanity.com/paper/2512.21576/) (2025)
- [Deep Thought Alignment: Trajectory-Level Latent Distillation for Video Reasoning](https://mlsanity.com/paper/2608.16316/) (2026)
- [ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs](https://mlsanity.com/paper/2507.21420/) (2025)
- [Robustness of Vision Language Models Against Split-Image Harmful Input Attacks](https://mlsanity.com/paper/2602.08136/) (2026)
- [Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models](https://mlsanity.com/paper/2603.21426/) (2026)
- [C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs](https://mlsanity.com/paper/2607.21076/) (2026)
- [When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs](https://mlsanity.com/paper/2509.16633/) (2025)
- [Text2Loc++: Generalizing 3D Point Cloud Localization from Natural Language](https://mlsanity.com/paper/2511.15308/) (2025)
- [When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA](https://mlsanity.com/paper/2511.17886/) (2025)
- [LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models](https://mlsanity.com/paper/2605.10641/) (2026)
- [Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation](https://mlsanity.com/paper/2607.02593/) (2026)
- [MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations](https://mlsanity.com/paper/2608.25575/) (2026)
- [MCD: Causal Distillation of Multimodal In-Context Learning in Large Vision-Language Models](https://mlsanity.com/paper/2609.39920/) (2026)
- [KVE-KD: Key Visual Evidence-Guided Knowledge Distillation for Vision-Language Models](https://mlsanity.com/paper/2610.03842/) (2026)

## Abstract (arXiv)

The recent surge in Multimodal Large Language Models (MLLMs) has showcased their remarkable potential for achieving generalized intelligence by integrating visual understanding into Large Language Models.Nevertheless, the sheer model size of MLLMs leads to substantial memory and computational demands that hinder their widespread deployment. In this work, we do not propose a new efficient model structure or train small-scale MLLMs from scratch. Instead, we focus on what matters for training small-scale MLLMs through knowledge distillation, which is the first step from the multimodal distillation perspective. Our extensive studies involve training strategies, model choices, and distillation algorithms in the knowledge distillation process. These results show that joint alignment for both tokens and logit alignment plays critical roles in teacher-student frameworks. In addition, we draw a series of intriguing observations from this study. By evaluating different benchmarks and proper strategy, even a 2.7B small-scale model can perform on par with larger models with 7B or 13B parameters. Our code and models will be publicly available for further research.