# XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning

arXiv: https://arxiv.org/abs/2211.13929 · HTML: https://mlsanity.com/paper/2211.13929/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 65 of its 79 references that are in the map.
- First posted on arXiv 25 Nov 2022.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 30 of 65)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language](https://mlsanity.com/paper/2202.03555/) (2022)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [End-to-End Learning of Visual Representations from Uncurated Instructional Videos](https://mlsanity.com/paper/1912.06430/) (2019)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Masked Autoencoders As Spatiotemporal Learners](https://mlsanity.com/paper/2205.09113/) (2022)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)

## Built on by (most cited first, 30 of 31)

- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restoration](https://mlsanity.com/paper/2602.23169/) (2026)
- [EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning](https://mlsanity.com/paper/2403.09502/) (2024)
- [Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation](https://mlsanity.com/paper/2410.11586/) (2024)
- [Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representations](https://mlsanity.com/paper/2507.03304/) (2025)
- [Learning Modality Knowledge Alignment for Cross-Modality Transfer](https://mlsanity.com/paper/2406.18864/) (2024)
- [DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learning](https://mlsanity.com/paper/2408.07080/) (2024)
- [Uncovering the Hidden Dynamics of Video Self-supervised Learning under Distribution Shifts](https://mlsanity.com/paper/2306.02014/) (2023)
- [Open-set Cross Modal Generalization via Multimodal Unified Representation](https://mlsanity.com/paper/2507.14935/) (2025)
- [Learning Task-Agnostic Representations through Multi-Teacher Distillation](https://mlsanity.com/paper/2510.18680/) (2025)
- [Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognition](https://mlsanity.com/paper/2408.09035/) (2024)
- [BaryIR: Learning Multi-Source Unified Representation in Continuous Barycenter Space for Generalizable All-in-One Image Restoration](https://mlsanity.com/paper/2505.21637/) (2025)
- [A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2412.00049/) (2024)
- [Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation](https://mlsanity.com/paper/2505.06803/) (2025)
- [A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features](https://mlsanity.com/paper/2505.21317/) (2025)
- [MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning](https://mlsanity.com/paper/2606.25225/) (2026)
- [Non-target Divergence Hypothesis: Toward Understanding Domain Gaps in Cross-Modal Knowledge Distillation](https://mlsanity.com/paper/2409.02438/) (2024)
- [ECG-guided individual identification via PPG](https://mlsanity.com/paper/2501.01983/) (2024)
- [Depth-Guided Self-Supervised Human Keypoint Detection via Cross-Modal Distillation](https://mlsanity.com/paper/2410.14700/) (2024)
- [Solution for 8th Competition on Affective & Behavior Analysis in-the-wild](https://mlsanity.com/paper/2503.11115/) (2025)
- [CA^2ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognition](https://mlsanity.com/paper/2503.23447/) (2025)
- [Continual Cross-Modal Generalization](https://mlsanity.com/paper/2504.00561/) (2025)
- [Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation](https://mlsanity.com/paper/2504.14307/) (2025)
- [Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning](https://mlsanity.com/paper/2510.13182/) (2025)
- [Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalization](https://mlsanity.com/paper/2602.03570/) (2026)
- [MuDD: A Multimodal Deception Detection Dataset and GSR-Guided Progressive Distillation for Non-Contact Deception Detection](https://mlsanity.com/paper/2603.26064/) (2026)
- [JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inference](https://mlsanity.com/paper/2603.29167/) (2026)
- [Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representations](https://mlsanity.com/paper/2605.12145/) (2026)
- [Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithm](https://mlsanity.com/paper/2606.10504/) (2026)
- [Cross-Modal Knowledge Distillation for Acoustic Pedestrian Detection](https://mlsanity.com/paper/2609.31785/) (2026)

## Abstract (arXiv)

We present XKD, a novel self-supervised framework to learn meaningful representations from unlabelled videos. XKD is trained with two pseudo objectives. First, masked data reconstruction is performed to learn modality-specific representations from audio and visual streams. Next, self-supervised cross-modal knowledge distillation is performed between the two modalities through a teacher-student setup to learn complementary information. We introduce a novel domain alignment strategy to tackle domain discrepancy between audio and visual modalities enabling effective cross-modal knowledge distillation. Additionally, to develop a general-purpose network capable of handling both audio and visual streams, modality-agnostic variants of XKD are introduced, which use the same pretrained backbone for different audio and visual tasks. Our proposed cross-modal knowledge distillation improves video action classification by 8% to 14% on UCF101, HMDB51, and Kinetics400. Additionally, XKD improves multimodal action classification by 5.5% on Kinetics-Sound. XKD shows state-of-the-art performance in sound classification on ESC50, achieving top-1 accuracy of 96.5%.