# Cross-Modal Knowledge Transfer Without Task-Relevant Source Data

arXiv: https://arxiv.org/abs/2209.04027 · HTML: https://mlsanity.com/paper/2209.04027/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 8 Sep 2022.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation](https://mlsanity.com/paper/2110.04202/) (2021)
- [Cross Modal Distillation for Supervision Transfer](https://mlsanity.com/paper/1507.00448/) (2015)
- [Generalized Source-free Domain Adaptation](https://mlsanity.com/paper/2108.01614/) (2021)
- [Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer](https://mlsanity.com/paper/2012.07297/) (2020)
- [Progressive Domain Adaptation for Object Detection](https://mlsanity.com/paper/1910.11319/) (2019)
- [Unsupervised Multi-source Domain Adaptation Without Access to Source Data](https://mlsanity.com/paper/2104.01845/) (2021)
- [Domain Adaptive Semantic Segmentation Using Weak Labels](https://mlsanity.com/paper/2007.15176/) (2020)
- [Cross-modal knowledge distillation for action recognition](https://mlsanity.com/paper/1910.04641/) (2019)
- [Zero-Shot Deep Domain Adaptation](https://mlsanity.com/paper/1707.01922/) (2017)
- [Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge](https://mlsanity.com/paper/2004.00176/) (2020)
- [Cross and Learn: Cross-Modal Self-Supervision](https://mlsanity.com/paper/1811.03879/) (2018)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognition](https://mlsanity.com/paper/1912.10982/) (2019)
- [Casting a BAIT for Offline and Online Source-free Domain Adaptation](https://mlsanity.com/paper/2010.12427/) (2020)
- [A Large RGB-D Dataset for Semi-supervised Monocular Depth Estimation](https://mlsanity.com/paper/1904.10230/) (2019)
- [Translate-to-Recognize Networks for RGB-D Scene Recognition](https://mlsanity.com/paper/1904.12254/) (2019)
- [Unsupervised Robust Domain Adaptation without Source Data](https://mlsanity.com/paper/2103.14577/) (2021)

## Built on by (most cited first, 17 of 17)

- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Source-Free Unsupervised Domain Adaptation: A Survey](https://mlsanity.com/paper/2301.00265/) (2022)
- [EventDance: Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition](https://mlsanity.com/paper/2403.14082/) (2024)
- [Effective Restoration of Source Knowledge in Continual Test Time Adaptation](https://mlsanity.com/paper/2311.04991/) (2023)
- [MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysis](https://mlsanity.com/paper/2308.16561/) (2023)
- [SUMMIT: Source-Free Adaptation of Uni-Modal Models to Multi-Modal Targets](https://mlsanity.com/paper/2308.11880/) (2023)
- [Is Extending Modality The Right Path Towards Omni-Modality?](https://mlsanity.com/paper/2506.01872/) (2025)
- [EventVGGT: Exploring Cross-Modal Distillation for Consistent Event-based Depth Estimation](https://mlsanity.com/paper/2603.09385/) (2026)
- [MAPS: A Noise-Robust Progressive Learning Approach for Source-Free Domain Adaptive Keypoint Detection](https://mlsanity.com/paper/2302.04589/) (2023)
- [Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Data](https://mlsanity.com/paper/2401.05014/) (2024)
- [A Mixed Diet Makes DINO An Omnivorous Vision Encoder](https://mlsanity.com/paper/2602.24181/) (2026)
- [Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection](https://mlsanity.com/paper/2309.16592/) (2023)
- [A Generalization Theory of Cross-Modality Distillation with Contrastive Learning](https://mlsanity.com/paper/2405.03355/) (2024)
- [EventDance++: Language-guided Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition](https://mlsanity.com/paper/2409.12778/) (2024)
- [COLA: Context-aware Language-driven Test-time Adaptation](https://mlsanity.com/paper/2509.17598/) (2025)
- [CONTRAST: Continual Multi-source Adaptation to Dynamic Distributions](https://mlsanity.com/paper/2401.02561/) (2024)
- [SFE-VGGT: Source-Free VGGT Distillation for Event-Based Monocular Depth Estimation](https://mlsanity.com/paper/2609.36929/) (2026)

## Abstract (arXiv)

Cost-effective depth and infrared sensors as alternatives to usual RGB sensors are now a reality, and have some advantages over RGB in domains like autonomous navigation and remote sensing. As such, building computer vision and deep learning systems for depth and infrared data are crucial. However, large labeled datasets for these modalities are still lacking. In such cases, transferring knowledge from a neural network trained on a well-labeled large dataset in the source modality (RGB) to a neural network that works on a target modality (depth, infrared, etc.) is of great value. For reasons like memory and privacy, it may not be possible to access the source data, and knowledge transfer needs to work with only the source models. We describe an effective solution, SOCKET: SOurce-free Cross-modal KnowledgE Transfer for this challenging task of transferring knowledge from one source modality to a different target modality without access to task-relevant source data. The framework reduces the modality gap using paired task-irrelevant data, as well as by matching the mean and variance of the target features with the batch-norm statistics that are present in the source models. We show through extensive experiments that our method significantly outperforms existing source-free methods for classification tasks which do not account for the modality gap.