Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement
On the map: Knowledge distillation → Multimodal distillation
Abstract (arXiv)
Knowledge distillation transfers knowledge from large teacher models to smaller students for efficient inference. While existing methods primarily focus on distillation strategies, they often overlook the importance of enhancing teacher knowledge quality. In this paper, we propose Text-guided Multi-view Knowledge Distillation (TMKD), which leverages dual-modality teachers, a visual teacher and a text teacher (CLIP), to provide richer supervisory signals. Specifically, we enhance the visual teacher with multi-view inputs incorporating visual priors (edge and high-frequency features), while the text teacher generates semantic weights through prior-aware prompts to guide adaptive feature fusion. Additionally, we introduce vision-language contrastive regularization to strengthen semantic knowledge in the student model. Extensive experiments on five benchmarks demonstrate that TMKD consistently improves knowledge distillation performance by up to 4.49%, validating the effectiveness of our dual-teacher multi-view enhancement strategy. Code is available at https://anonymous.4open.science/r/TMKD-main-44D1.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 18 of its 40 references that are in the map. First posted on arXiv 25 Mar 2026.
Builds on
18 of 40 references.
Closest ancestors · 1
Same direction, newest first
Other references · 17
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2020Knowledge Distillation: A Surveycited by 1,603
- 2019Relational Knowledge Distillationcited by 889
- 2019Contrastive Representation Distillationcited by 748
- 2019Similarity-Preserving Knowledge Distillationcited by 516
- 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 416
- 2022Decoupled Knowledge Distillationcited by 388
- 2023Class Attention Transfer Based Knowledge Distillationcited by 61
- 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
- 2024Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detectioncited by 26
- 2024MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolutioncited by 17
5 more
- 2023Adaptive Multi-Teacher Knowledge Distillation with Meta-Learningcited by 16
- 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognitioncited by 16
- 2024Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Expertscited by 14
- 2023Dual Relation Knowledge Distillation for Object Detectioncited by 9
- 2024Noisy Node Classification by Bi-level Optimization based Multi-teacher Distillationcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
- 2025Enriching Knowledge Distillation with Cross-Modal Teacher Fusion
- 2025When Better Teachers Don't Make Better Students: Revisiting Knowledge Distillation for CLIP Models in VQA
- 2026Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations
- 2025Perspective-Aware Teaching: Adapting Knowledge for Heterogeneous Distillation
- 2026KVE-KD: Key Visual Evidence-Guided Knowledge Distillation for Vision-Language Models
- 2026LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models
- 2025Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity