mlsanityData to 14 Oct 2026

XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning

arXiv 2211.13929 ↗First posted 25 Nov 2022cs.CVcited by 31

On the map: Knowledge distillation → Multimodal distillation

Abstract (arXiv)

We present XKD, a novel self-supervised framework to learn meaningful representations from unlabelled videos. XKD is trained with two pseudo objectives. First, masked data reconstruction is performed to learn modality-specific representations from audio and visual streams. Next, self-supervised cross-modal knowledge distillation is performed between the two modalities through a teacher-student setup to learn complementary information. We introduce a novel domain alignment strategy to tackle domain discrepancy between audio and visual modalities enabling effective cross-modal knowledge distillation. Additionally, to develop a general-purpose network capable of handling both audio and visual streams, modality-agnostic variants of XKD are introduced, which use the same pretrained backbone for different audio and visual tasks. Our proposed cross-modal knowledge distillation improves video action classification by 8% to 14% on UCF101, HMDB51, and Kinetics400. Additionally, XKD improves multimodal action classification by 5.5% on Kinetics-Sound. XKD shows state-of-the-art performance in sound classification on ESC50, achieving top-1 accuracy of 96.5%.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Multimodal distillation, one of 12 directions in Knowledge distillation. Builds on 65 of its 79 references that are in the map. First posted on arXiv 25 Nov 2022.

Builds on

65 of 79 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection8 Aug

Other references · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
38 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  2. 2017The Kinetics Human Action Video Datasetcited by 1,909
  3. 2021ViViT: A Video Vision Transformercited by 1,620
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  6. 2017Mixed Precision Trainingcited by 1,156
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  8. 2019On the Convergence of Adam and Beyondcited by 841
  9. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2021AST: Audio Spectrogram Transformercited by 643
  12. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  13. 2017Look, Listen and Learncited by 458
  14. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  15. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  16. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  17. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  18. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  19. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  20. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  21. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 267
  22. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 249
  23. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 247
  24. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  25. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  26. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  27. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  28. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 195
  29. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  30. 2018Self-Supervised Video Representation Learning with Space-Time Cubic Puzzlescited by 149
  31. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 146
  32. 2021Broaden Your Views for Self-Supervised Video Learningcited by 108
  33. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
  34. 2019ASR is all you need: cross-modal distillation for lip readingcited by 88
  35. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  36. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 82
  37. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  38. 2021Robust Audio-Visual Instance Discriminationcited by 80

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 9; 2025: 11; 2026: 9.

31 papers.

Sort:

By area:

  1. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 51
  2. 2026Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restorationcited by 15
  3. 2024EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learningcited by 12
  4. 2024Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillationcited by 11
  5. 2025Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representationscited by 9
  6. 2024Learning Modality Knowledge Alignment for Cross-Modality Transfercited by 8
  7. 2024DisCoM-KD: Cross-Modal Knowledge Distillation via Disentanglement Representation and Adversarial Learningcited by 6
  8. 2023Uncovering the Hidden Dynamics of Video Self-supervised Learning under Distribution Shiftscited by 4
  9. 2025Open-set Cross Modal Generalization via Multimodal Unified Representationcited by 4
  10. 2025Learning Task-Agnostic Representations through Multi-Teacher Distillationcited by 4
  11. 2024Multi Teacher Privileged Knowledge Distillation for Multimodal Expression Recognitioncited by 3
  12. 2025BaryIR: Learning Multi-Source Unified Representation in Continuous Barycenter Space for Generalizable All-in-One Image Restorationcited by 3
19 more
  1. 2024A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learningcited by 2
  2. 2025Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillationcited by 2
  3. 2025A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Featurescited by 2
  4. 2026MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learningcited by 2
  5. 2024Non-target Divergence Hypothesis: Toward Understanding Domain Gaps in Cross-Modal Knowledge Distillationcited by 1
  6. 2024ECG-guided individual identification via PPGcited by 1
  7. 2024Depth-Guided Self-Supervised Human Keypoint Detection via Cross-Modal Distillationcited by 0
  8. 2025Solution for 8th Competition on Affective & Behavior Analysis in-the-wildcited by 0
  9. 2025CA^2ST: Cross-Attention in Audio, Space, and Time for Holistic Video Recognitioncited by 0
  10. 2025Continual Cross-Modal Generalizationcited by 0
  11. 2025Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillationcited by 0
  12. 2025Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learningcited by 0
  13. 2026Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalizationcited by 0
  14. 2026MuDD: A Multimodal Deception Detection Dataset and GSR-Guided Progressive Distillation for Non-Contact Deception Detectioncited by 0
  15. 2026JDCNet: Confidence-Gated Privileged-Modality Distillation for Cost-Preserving X-ray Inferencecited by 0
  16. 2026Cross-Modal-Domain Generalization Through Semantically Aligned Discrete Representationscited by 0
  17. 2026Cross-Modal Knowledge Distillation without Paired Data: Theoretical Foundation and Algorithmcited by 0
  18. 2026Cross-Modal Knowledge Distillation for Acoustic Pedestrian Detectioncited by 0
  19. 2026LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Robust Cross-Modal Knowledge Distillation for Unconstrained Videos
  2. 2025View-aware Cross-modal Distillation for Multi-view Action Recognition
  3. 2023Enhanced Multimodal Representation Learning with Cross-modal KD
  4. 2021Cross-Modal Discrete Representation Learning
  5. 2022CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation
  6. 2023VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning
  7. 2025Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models
  8. 2025Distilling Cross-Modal Knowledge via Feature Disentanglement