# Powerful Teachers Matter: Text-Guided Multi-view Knowledge Distillation with Visual Prior Enhancement

arXiv: https://arxiv.org/abs/2603.24208 · HTML: https://mlsanity.com/paper/2603.24208/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 40 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [Class Attention Transfer Based Knowledge Distillation](https://mlsanity.com/paper/2304.12777/) (2023)
- [Knowledge Distillation for Federated Learning: a Practical Guide](https://mlsanity.com/paper/2211.04742/) (2022)
- [Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection](https://mlsanity.com/paper/2401.15123/) (2024)
- [MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution](https://mlsanity.com/paper/2404.09571/) (2024)
- [Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning](https://mlsanity.com/paper/2306.06634/) (2023)
- [Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition](https://mlsanity.com/paper/2502.18510/) (2025)
- [Lifelong Knowledge Editing for Vision Language Models with Low-Rank Mixture-of-Experts](https://mlsanity.com/paper/2411.15432/) (2024)
- [Dual Relation Knowledge Distillation for Object Detection](https://mlsanity.com/paper/2302.05637/) (2023)
- [Noisy Node Classification by Bi-level Optimization based Multi-teacher Distillation](https://mlsanity.com/paper/2404.17875/) (2024)
- [Multi-aspect Knowledge Distillation with Large Language Model](https://mlsanity.com/paper/2501.13341/) (2025)

## Abstract (arXiv)

Knowledge distillation transfers knowledge from large teacher models to smaller students for efficient inference. While existing methods primarily focus on distillation strategies, they often overlook the importance of enhancing teacher knowledge quality. In this paper, we propose Text-guided Multi-view Knowledge Distillation (TMKD), which leverages dual-modality teachers, a visual teacher and a text teacher (CLIP), to provide richer supervisory signals. Specifically, we enhance the visual teacher with multi-view inputs incorporating visual priors (edge and high-frequency features), while the text teacher generates semantic weights through prior-aware prompts to guide adaptive feature fusion. Additionally, we introduce vision-language contrastive regularization to strengthen semantic knowledge in the student model. Extensive experiments on five benchmarks demonstrate that TMKD consistently improves knowledge distillation performance by up to 4.49%, validating the effectiveness of our dual-teacher multi-view enhancement strategy. Code is available at https://anonymous.4open.science/r/TMKD-main-44D1.