# Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge

arXiv: https://arxiv.org/abs/2004.00176 · HTML: https://mlsanity.com/paper/2004.00176/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2020.

## Built on by (most cited first, 30 of 35)

- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [FSDR: Frequency Space Domain Randomization for Domain Generalization](https://mlsanity.com/paper/2103.02370/) (2021)
- [2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds](https://mlsanity.com/paper/2207.04397/) (2022)
- [Parser-Free Virtual Try-on via Distilling Appearance Flows](https://mlsanity.com/paper/2103.04559/) (2021)
- [LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detector](https://mlsanity.com/paper/2108.08258/) (2021)
- [X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioning](https://mlsanity.com/paper/2203.00843/) (2022)
- [EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation](https://mlsanity.com/paper/2111.12341/) (2021)
- [Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction](https://mlsanity.com/paper/2109.01723/) (2021)
- [ToAlign: Task-oriented Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2106.10812/) (2021)
- [Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translation](https://mlsanity.com/paper/2109.01801/) (2021)
- [Hand Image Understanding via Deep Multi-Task Learning](https://mlsanity.com/paper/2107.11646/) (2021)
- [SFD2: Semantic-guided Feature Detection and Description](https://mlsanity.com/paper/2304.14845/) (2023)
- [Multimodal Knowledge Expansion](https://mlsanity.com/paper/2103.14431/) (2021)
- [EventDance: Unsupervised Source-free Cross-modal Adaptation for Event-based Object Recognition](https://mlsanity.com/paper/2403.14082/) (2024)
- [Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognition](https://mlsanity.com/paper/2308.11489/) (2023)
- [OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of All](https://mlsanity.com/paper/2405.16108/) (2024)
- [Out-of-Domain Generalization from a Single Source: An Uncertainty Quantification Approach](https://mlsanity.com/paper/2108.02888/) (2021)
- [Cross-Modal Knowledge Transfer Without Task-Relevant Source Data](https://mlsanity.com/paper/2209.04027/) (2022)
- [HandTailor: Towards High-Precision Monocular 3D Hand Recovery](https://mlsanity.com/paper/2102.09244/) (2021)
- [StructVPR: Distill Structural Knowledge with Weighting Samples for Visual Place Recognition](https://mlsanity.com/paper/2212.00937/) (2022)
- [Decomposed Cross-modal Distillation for RGB-based Temporal Action Detection](https://mlsanity.com/paper/2303.17285/) (2023)
- [Breaking Modality Gap in RGBT Tracking: Coupled Knowledge Distillation](https://mlsanity.com/paper/2410.11586/) (2024)
- [Applications of Knowledge Distillation in Remote Sensing: A Survey](https://mlsanity.com/paper/2409.12111/) (2024)
- [MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysis](https://mlsanity.com/paper/2308.16561/) (2023)
- [MTMMC: A Large-Scale Real-World Multi-Modal Camera Tracking Benchmark](https://mlsanity.com/paper/2403.20225/) (2024)
- [StructVPR++: Distill Structural and Semantic Knowledge with Weighting Samples for Visual Place Recognition](https://mlsanity.com/paper/2503.06601/) (2025)
- [A hybrid classification-regression approach for 3D hand pose estimation using graph convolutional networks](https://mlsanity.com/paper/2105.10902/) (2021)
- [Iterative Self Knowledge Distillation -- From Pothole Classification to Fine-Grained and COVID Recognition](https://mlsanity.com/paper/2202.02265/) (2022)
- [Practical Knowledge Distillation: Using DNNs to Beat DNNs](https://mlsanity.com/paper/2302.12360/) (2023)
- [A Generalization Theory of Cross-Modality Distillation with Contrastive Learning](https://mlsanity.com/paper/2405.03355/) (2024)

## Abstract (arXiv)

Cross-modal knowledge distillation deals with transferring knowledge from a model trained with superior modalities (Teacher) to another model trained with weak modalities (Student). Existing approaches require paired training examples exist in both modalities. However, accessing the data from superior modalities may not always be feasible. For example, in the case of 3D hand pose estimation, depth maps, point clouds, or stereo images usually capture better hand structures than RGB images, but most of them are expensive to be collected. In this paper, we propose a novel scheme to train the Student in a Target dataset where the Teacher is unavailable. Our key idea is to generalize the distilled cross-modal knowledge learned from a Source dataset, which contains paired examples from both modalities, to the Target dataset by modeling knowledge as priors on parameters of the Student. We name our method "Cross-Modal Knowledge Generalization" and demonstrate that our scheme results in competitive performance for 3D hand pose estimation on standard benchmark datasets.