mlsanityData to 7 Oct 2026

MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders

arXiv 2501.01709 ↗First posted 3 Jan 2025cs.CV cs.AIcited by 25

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Visual encoders are fundamental components in vision-language models (VLMs), each showcasing unique strengths derived from various pre-trained visual foundation models. To leverage the various capabilities of these encoders, recent studies incorporate multiple encoders within a single VLM, leading to a considerable increase in computational cost. In this paper, we present Mixture-of-Visual-Encoder Knowledge Distillation (MoVE-KD), a novel framework that distills the unique proficiencies of multiple vision encoders into a single, efficient encoder model. Specifically, to mitigate conflicts and retain the unique characteristics of each teacher encoder, we employ low-rank adaptation (LoRA) and mixture-of-experts (MoEs) to selectively activate specialized knowledge based on input features, enhancing both adaptability and efficiency. To regularize the KD process and enhance performance, we propose an attention-based distillation strategy that adaptively weighs the different encoders and emphasizes valuable visual tokens, reducing the burden of replicating comprehensive but distinct features from multiple teachers. Comprehensive experiments on popular VLMs, such as LLaVA and LLaVA-NeXT, validate the effectiveness of our method. Our code is available at: https://github.com/hey-cjj/MoVE-KD.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 55 references that are in the map. First posted on arXiv 3 Jan 2025.

Builds on

45 of 55 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  2. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation28 Aug
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  4. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts18 May
  5. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  6. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2023Qwen Technical Reportcited by 3,273
23 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  7. 2023Vision Transformers Need Registerscited by 844
  8. 2024Yi: Open Foundation Models by 01.AIcited by 706
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  10. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  12. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  13. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  14. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  15. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  16. 2022Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillationcited by 93
  17. 2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 75
  18. 2024When Do We Not Need Larger Vision Models?cited by 65
  19. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  20. 2024LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Modelcited by 43
  21. 2022One Student Knows All Experts Know: From Sparse to Densecited by 25
  22. 2022Masked Distillation with Receptive Tokenscited by 23
  23. 2024Does Combining Parameter-efficient Modules Improve Few-shot Transfer Accuracy?cited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 13.

25 papers.

Sort:

By area:

  1. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  2. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 69
  3. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 24
  4. 2025FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruningcited by 20
  5. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  6. 2026HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensingcited by 9
  7. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  8. 2025Distilling Cross-Modal Knowledge via Feature Disentanglementcited by 5
  9. 2026Gated Relational Alignment via Confidence-based Distillation for Efficient VLMscited by 5
  10. 2026Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactionscited by 5
  11. 2025Concept-as-Tree: A Controllable Synthetic Data Framework Makes Stronger Personalized VLMscited by 4
  12. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Modelscited by 4
13 more
  1. 2025EM-KD: Distilling Efficient Multimodal Large Language Model with Unbalanced Vision Tokenscited by 4
  2. 2026EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillationcited by 3
  3. 2026How Well Do Vision-Language Models Understand Sequential Driving Scenes? A Sensitivity Studycited by 2
  4. 2025ActVAR: Activating Mixtures of Weights and Tokens for Efficient Visual Autoregressive Generationcited by 1
  5. 2025NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devicescited by 0
  6. 2026Drive-KD: Multi-Teacher Distillation for VLMs in Autonomous Drivingcited by 0
  7. 2026Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Modelscited by 0
  8. 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Modelscited by 0
  9. 2026Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMscited by 0
  10. 2026FADA: Accessible fetal ultrasound interpretation and annotation with a selectively distilled unified vision-language modelcited by 0
  11. 2026Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillationcited by 0
  12. 2026COSMO: Consensus-Driven Shift Modulation for Source-Free Domain Adaptationcited by 0
  13. 2026When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition
  2. 2026DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer
  3. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
  4. 2026KVE-KD: Key Visual Evidence-Guided Knowledge Distillation for Vision-Language Models
  5. 2025Simple yet Effective Semi-supervised Knowledge Distillation from Vision-Language Models via Dual-Head Optimization
  6. 2024Select and Distill: Selective Dual-Teacher Knowledge Transfer for Continual Learning on Vision-Language Models
  7. 2025MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processing
  8. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey