# Enriching Knowledge Distillation with Cross-Modal Teacher Fusion

arXiv: https://arxiv.org/abs/2511.09286 · HTML: https://mlsanity.com/paper/2511.09286/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- First posted on arXiv 12 Nov 2025.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Built on by (most cited first, 2 of 2)

- [Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding](https://mlsanity.com/paper/2603.11423/) (2026)
- [Multi-Modal Anomaly Detection: A Survey](https://mlsanity.com/paper/2608.24937/) (2026)

## Abstract (arXiv)

Multi-teacher knowledge distillation (KD), a more effective technique than traditional single-teacher methods, transfers knowledge from expert teachers to a compact student model using logit or feature matching. However, most existing approaches lack knowledge diversity, as they rely solely on unimodal visual information, overlooking the potential of cross-modal representations. In this work, we explore the use of CLIP's vision-language knowledge as a complementary source of supervision for KD, an area that remains largely underexplored. We propose a simple yet effective framework that fuses the logits and features of a conventional teacher with those from CLIP. By incorporating CLIP's multi-prompt textual guidance, the fused supervision captures both dataset-specific and semantically enriched visual cues. Beyond accuracy, analysis shows that the fused teacher yields more confident and reliable predictions, significantly increasing confident-correct cases while reducing confidently wrong ones. Moreover, fusion with CLIP refines the entire logit distribution, producing semantically meaningful probabilities for non-target classes, thereby improving inter-class consistency and distillation quality. Despite its simplicity, the proposed method, Enriching Knowledge Distillation (RichKD), consistently outperforms most existing baselines across multiple benchmarks and exhibits stronger robustness under distribution shifts and input corruptions.