# Distilling Cross-Modal Knowledge via Feature Disentanglement

arXiv: https://arxiv.org/abs/2511.19887 · HTML: https://mlsanity.com/paper/2511.19887/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 25 Nov 2025.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [Learning in the Frequency Domain](https://mlsanity.com/paper/2002.12416/) (2020)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [FastDepth: Fast Monocular Depth Estimation on Embedded Systems](https://mlsanity.com/paper/1903.03273/) (2019)
- [CrisisMMD: Multimodal Twitter Datasets from Natural Disasters](https://mlsanity.com/paper/1805.00713/) (2018)
- [DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks](https://mlsanity.com/paper/1901.09229/) (2019)
- [Cross-modal knowledge distillation for action recognition](https://mlsanity.com/paper/1910.04641/) (2019)
- [Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2310.01035/) (2023)
- [The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation](https://mlsanity.com/paper/2206.06487/) (2022)
- [Enhancing multimodal cooperation via sample-level modality valuation](https://mlsanity.com/paper/2309.06255/) (2023)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [CRKD: Enhanced Camera-Radar Object Detection with Cross-modality Knowledge Distillation](https://mlsanity.com/paper/2403.19104/) (2024)
- [FreeKD: Knowledge Distillation via Semantic Frequency Prompt](https://mlsanity.com/paper/2311.12079/) (2023)
- [MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders](https://mlsanity.com/paper/2501.01709/) (2025)
- [FastDriveVLA: Efficient End-to-End Driving via Plug-and-Play Reconstruction-based Token Pruning](https://mlsanity.com/paper/2507.23318/) (2025)
- [Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertainty](https://mlsanity.com/paper/2305.02722/) (2023)

## Built on by (most cited first, 5 of 5)

- [UniCSG: Unified High-Fidelity Content-Constrained Style-Driven Generation via Staged Semantic and Frequency Disentanglement](https://mlsanity.com/paper/2604.17850/) (2026)
- [FreqKD: Frequency-Decoupled Cross-Modal Knowledge Distillation for Infrared Object Detection](https://mlsanity.com/paper/2606.11572/) (2026)
- [FD-CanKD: Frequency-Decoupled Cross-Attention Distillation as a Refinement Prior for Compact Object Detectors](https://mlsanity.com/paper/2608.18590/) (2026)
- [Multi-Modal Anomaly Detection: A Survey](https://mlsanity.com/paper/2608.24937/) (2026)
- [Codebook-Guided Cross-Modal Knowledge Distillation for Structurally Heterogeneous Features](https://mlsanity.com/paper/2609.37243/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) has proven highly effective for compressing large models and enhancing the performance of smaller ones. However, its effectiveness diminishes in cross-modal scenarios, such as vision-to-language distillation, where inconsistencies in representation across modalities lead to difficult knowledge transfer. To address this challenge, we propose frequency-decoupled cross-modal knowledge distillation, a method designed to decouple and balance knowledge transfer across modalities by leveraging frequency-domain features. We observed that low-frequency features exhibit high consistency across different modalities, whereas high-frequency features demonstrate extremely low cross-modal similarity. Accordingly, we apply distinct losses to these features: enforcing strong alignment in the low-frequency domain and introducing relaxed alignment for high-frequency features. We also propose a scale consistency loss to address distributional shifts between modalities, and employ a shared classifier to unify feature spaces. Extensive experiments across multiple benchmark datasets show our method substantially outperforms traditional KD and state-of-the-art cross-modal KD approaches. Code is available at https://github.com/Johumliu/FD-CMKD.