# MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders

arXiv: https://arxiv.org/abs/2501.01709 · HTML: https://mlsanity.com/paper/2501.01709/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 45 of its 55 references that are in the map.
- First posted on arXiv 3 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference](https://mlsanity.com/paper/2410.04417/) (2024)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)

## Built on by (most cited first, 26 of 26)

- [RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete](https://mlsanity.com/paper/2502.21257/) (2025)
- [MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation](https://mlsanity.com/paper/2503.20384/) (2025)
- [StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression](https://mlsanity.com/paper/2511.07278/) (2025)
- [FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning](https://mlsanity.com/paper/2507.23318/) (2025)
- [GenRecal: Generation after Recalibration from Large to Small Vision-Language Models](https://mlsanity.com/paper/2506.15681/) (2025)
- [HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensing](https://mlsanity.com/paper/2603.15257/) (2026)
- [Masking Teacher and Reinforcing Student for Distilling Vision-Language Models](https://mlsanity.com/paper/2512.22238/) (2025)
- [Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs](https://mlsanity.com/paper/2601.22709/) (2026)
- [HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models](https://mlsanity.com/paper/2506.19072/) (2025)
- [Distilling Cross-Modal Knowledge via Feature Disentanglement](https://mlsanity.com/paper/2511.19887/) (2025)
- [EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokens](https://mlsanity.com/paper/2511.21106/) (2025)
- [Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions](https://mlsanity.com/paper/2602.09483/) (2026)
- [Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMs](https://mlsanity.com/paper/2503.12999/) (2025)
- [EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillation](https://mlsanity.com/paper/2603.09465/) (2026)
- [How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Study](https://mlsanity.com/paper/2604.06750/) (2026)
- [Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models](https://mlsanity.com/paper/2604.14629/) (2026)
- [ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generation](https://mlsanity.com/paper/2511.12893/) (2025)
- [NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devices](https://mlsanity.com/paper/2504.03118/) (2025)
- [Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Driving](https://mlsanity.com/paper/2601.21288/) (2026)
- [LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models](https://mlsanity.com/paper/2605.10641/) (2026)
- [Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs](https://mlsanity.com/paper/2606.03879/) (2026)
- [FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language model](https://mlsanity.com/paper/2606.11106/) (2026)
- [Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation](https://mlsanity.com/paper/2607.02593/) (2026)
- [COSMO: Consensus-Driven Shift Modulation for Source-Free Domain Adaptation](https://mlsanity.com/paper/2608.04604/) (2026)
- [When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Model](https://mlsanity.com/paper/2609.34861/) (2026)
- [GRAFT: Growing Agglomerative Foundation Models via Continual Teacher Distillation](https://mlsanity.com/paper/2610.02597/) (2026)

## Abstract (arXiv)

Visual encoders are fundamental components in vision-language models (VLMs), each showcasing unique strengths derived from various pre-trained visual foundation models. To leverage the various capabilities of these encoders, recent studies incorporate multiple encoders within a single VLM, leading to a considerable increase in computational cost. In this paper, we present Mixture-of-Visual-Encoder Knowledge Distillation (MoVE-KD), a novel framework that distills the unique proficiencies of multiple vision encoders into a single, efficient encoder model. Specifically, to mitigate conflicts and retain the unique characteristics of each teacher encoder, we employ low-rank adaptation (LoRA) and mixture-of-experts (MoEs) to selectively activate specialized knowledge based on input features, enhancing both adaptability and efficiency. To regularize the KD process and enhance performance, we propose an attention-based distillation strategy that adaptively weighs the different encoders and emphasizes valuable visual tokens, reducing the burden of replicating comprehensive but distinct features from multiple teachers. Comprehensive experiments on popular VLMs, such as LLaVA and LLaVA-NeXT, validate the effectiveness of our method. Our code is available at: https://github.com/hey-cjj/MoVE-KD.