# Dynamic Distillation Network for Cross-Domain Few-Shot Recognition with Unlabeled Data

arXiv: https://arxiv.org/abs/2106.07807 · HTML: https://mlsanity.com/paper/2106.07807/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 29 of 29)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases](https://mlsanity.com/paper/1705.02315/) (2017)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1902.03368/) (2019)
- [Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning](https://mlsanity.com/paper/1908.02983/) (2019)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [Interpolation Consistency Training for Semi-Supervised Learning](https://mlsanity.com/paper/1903.03825/) (2019)
- [Billion-scale semi-supervised learning for image classification](https://mlsanity.com/paper/1905.00546/) (2019)
- [Using Deep Learning for Image-Based Plant Disease Detection](https://mlsanity.com/paper/1604.03169/) (2016)
- [Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning](https://mlsanity.com/paper/1805.10002/) (2018)
- [Edge-labeling Graph Neural Network for Few-shot Learning](https://mlsanity.com/paper/1905.01436/) (2019)
- [Iterative Pseudo-Labeling for Speech Recognition](https://mlsanity.com/paper/2005.09267/) (2020)
- [A Broad Study on the Transferability of Visual Representations with Contrastive Learning](https://mlsanity.com/paper/2103.13517/) (2021)
- [Self-training for Few-shot Transfer Across Extreme Task Differences](https://mlsanity.com/paper/2010.07734/) (2020)
- [DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning](https://mlsanity.com/paper/1810.05726/) (2018)
- [Extending and Analyzing Self-Supervised Learning Across Domains](https://mlsanity.com/paper/2004.11992/) (2020)
- [KaoKore: A Pre-modern Japanese Art Facial Expression Dataset](https://mlsanity.com/paper/2002.08595/) (2020)

## Built on by (most cited first, 30 of 50)

- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2302.09309/) (2023)
- [Understanding Cross-Domain Few-Shot Learning Based on Domain Similarity and Few-Shot Difficulty](https://mlsanity.com/paper/2202.01339/) (2022)
- [Wave-SAN: Wavelet based Style Augmentation Network for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2203.07656/) (2022)
- [Context-Aware Meta-Learning](https://mlsanity.com/paper/2310.10971/) (2023)
- [CD-FSOD: A Benchmark for Cross-domain Few-shot Object Detection](https://mlsanity.com/paper/2210.05311/) (2022)
- [Dual Adaptive Representation Alignment for Cross-domain Few-shot Learning](https://mlsanity.com/paper/2306.10511/) (2023)
- [Deep Learning for Cross-Domain Few-Shot Visual Recognition: A Survey](https://mlsanity.com/paper/2303.08557/) (2023)
- [Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognition](https://mlsanity.com/paper/2405.19917/) (2024)
- [CDFSL-V: Cross-Domain Few-Shot Learning for Videos](https://mlsanity.com/paper/2309.03989/) (2023)
- [Enhancing Information Maximization with Distance-Aware Contrastive Learning for Source-Free Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2403.01966/) (2024)
- [Order-preserving Consistency Regularization for Domain Adaptation and Generalization](https://mlsanity.com/paper/2309.13258/) (2023)
- [Prompt as Free Lunch: Enhancing Diversity in Source-Free Cross-domain Few-shot Learning through Semantic-Guided Prompting](https://mlsanity.com/paper/2412.00767/) (2024)
- [ME-D2N: Multi-Expert Domain Decompositional Network for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2210.05280/) (2022)
- [Cross-Domain HAR: Few Shot Transfer Learning for Human Activity Recognition](https://mlsanity.com/paper/2310.14390/) (2023)
- [Direct Distillation between Different Domains](https://mlsanity.com/paper/2401.06826/) (2024)
- [Taxonomy Adaptive Cross-Domain Adaptation in Medical Imaging via Optimization Trajectory Distillation](https://mlsanity.com/paper/2307.14709/) (2023)
- [Cross-Domain Few-Shot Learning via Adaptive Transformer Networks](https://mlsanity.com/paper/2401.13987/) (2024)
- [ReFine: Re-randomization before Fine-tuning for Cross-domain Few-shot Learning](https://mlsanity.com/paper/2205.05282/) (2022)
- [How to Fine-tune Models with Few Samples: Update, Data Augmentation, and Test-time Augmentation](https://mlsanity.com/paper/2205.07874/) (2022)
- [Robust Domain Adaptive Object Detection with Unified Multi-Granularity Alignment](https://mlsanity.com/paper/2301.00371/) (2023)
- [Cross-Domain Cross-Set Few-Shot Learning via Learning Compact and Aligned Representations](https://mlsanity.com/paper/2207.07826/) (2022)
- [Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models](https://mlsanity.com/paper/2309.01479/) (2023)
- [Towards Few-Shot Learning in the Open World: A Review and Beyond](https://mlsanity.com/paper/2408.09722/) (2024)
- [Step-wise Distribution Alignment Guided Style Prompt Tuning for Source-free Cross-domain Few-shot Learning](https://mlsanity.com/paper/2411.10070/) (2024)
- [Self-Taught Cross-Domain Few-Shot Learning with Weakly Supervised Object Localization and Task-Decomposition](https://mlsanity.com/paper/2109.01302/) (2021)
- [Collaborating Domain-shared and Target-specific Feature Clustering for Cross-domain 3D Action Recognition](https://mlsanity.com/paper/2207.09767/) (2022)
- [Cross-Domain Few-Shot Classification via Inter-Source Stylization](https://mlsanity.com/paper/2208.08015/) (2022)
- [Distilling from Similar Tasks for Transfer Learning on a Budget](https://mlsanity.com/paper/2304.12314/) (2023)
- [Adaptive Parametric Prototype Learning for Cross-Domain Few-Shot Classification](https://mlsanity.com/paper/2309.01342/) (2023)

## Abstract (arXiv)

Most existing works in few-shot learning rely on meta-learning the network on a large base dataset which is typically from the same domain as the target dataset. We tackle the problem of cross-domain few-shot learning where there is a large shift between the base and target domain. The problem of cross-domain few-shot recognition with unlabeled target data is largely unaddressed in the literature. STARTUP was the first method that tackles this problem using self-training. However, it uses a fixed teacher pretrained on a labeled base dataset to create soft labels for the unlabeled target samples. As the base dataset and unlabeled dataset are from different domains, projecting the target images in the class-domain of the base dataset with a fixed pretrained model might be sub-optimal. We propose a simple dynamic distillation-based approach to facilitate unlabeled images from the novel/base dataset. We impose consistency regularization by calculating predictions from the weakly-augmented versions of the unlabeled images from a teacher network and matching it with the strongly augmented versions of the same images from a student network. The parameters of the teacher network are updated as exponential moving average of the parameters of the student network. We show that the proposed network learns representation that can be easily adapted to the target domain even though it has not been trained with target-specific classes during the pretraining phase. Our model outperforms the current state-of-the art method by 4.4% for 1-shot and 3.6% for 5-shot classification in the BSCD-FSL benchmark, and also shows competitive performance on traditional in-domain few-shot learning task.