# Self-supervised Knowledge Distillation for Few-shot Learning

arXiv: https://arxiv.org/abs/2006.09785 · HTML: https://mlsanity.com/paper/2006.09785/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2020.

## Built on by (most cited first, 30 of 41)

- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning](https://mlsanity.com/paper/2103.01315/) (2021)
- [Orthogonal Projection Loss](https://mlsanity.com/paper/2103.14021/) (2021)
- [Temperature as Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2110.04403/) (2021)
- [LibFewShot: A Comprehensive Library for Few-shot Learning](https://mlsanity.com/paper/2109.04898/) (2021)
- [Does Few-shot Learning Suffer from Backdoor Attacks?](https://mlsanity.com/paper/2401.01377/) (2023)
- [Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model](https://mlsanity.com/paper/2310.17653/) (2023)
- [Few-Shot Learning with a Strong Teacher](https://mlsanity.com/paper/2107.00197/) (2021)
- [ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time](https://mlsanity.com/paper/2206.15049/) (2022)
- [Rich Semantics Improve Few-shot Learning](https://mlsanity.com/paper/2104.12709/) (2021)
- [MUSCLE: A Model Update Strategy for Compatible LLM Evolution](https://mlsanity.com/paper/2407.09435/) (2024)
- [Squeezing Backbone Feature Distributions to the Max for Efficient Few-Shot Learning](https://mlsanity.com/paper/2110.09446/) (2021)
- [Towards Data-Free Domain Generalization](https://mlsanity.com/paper/2110.04545/) (2021)
- [Label Hallucination for Few-Shot Classification](https://mlsanity.com/paper/2112.03340/) (2021)
- [What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data](https://mlsanity.com/paper/2106.06410/) (2021)
- [Cap2Aug: Caption guided Image to Image data Augmentation](https://mlsanity.com/paper/2212.05404/) (2022)
- [ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback](https://mlsanity.com/paper/2107.14035/) (2021)
- [Revisiting Pool-based Prompt Learning for Few-shot Class-incremental Learning](https://mlsanity.com/paper/2507.09183/) (2025)
- [ESPT: A Self-Supervised Episodic Spatial Pretext Task for Improving Few-Shot Learning](https://mlsanity.com/paper/2304.13287/) (2023)
- [Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers](https://mlsanity.com/paper/2404.09326/) (2024)
- [The Art of Camouflage: Few-Shot Learning for Animal Detection and Segmentation](https://mlsanity.com/paper/2304.07444/) (2023)
- [MetaCoCo: A New Few-Shot Classification Benchmark with Spurious Correlation](https://mlsanity.com/paper/2404.19644/) (2024)
- [Distilling the Undistillable: Learning from a Nasty Teacher](https://mlsanity.com/paper/2210.11728/) (2022)
- [Guided Deep Metric Learning](https://mlsanity.com/paper/2206.02029/) (2022)
- [Visual-Semantic Contrastive Alignment for Few-Shot Image Classification](https://mlsanity.com/paper/2210.11000/) (2022)
- [Learning to Scale Temperature in Masked Self-Attention for Image Inpainting](https://mlsanity.com/paper/2302.06130/) (2023)
- [Few-shot Classification via Ensemble Learning with Multi-Order Statistics](https://mlsanity.com/paper/2305.00454/) (2023)
- [Leveraging Normalization Layer in Adapters With Progressive Learning and Adaptive Distillation for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2312.11260/) (2023)
- [Enabling the Network to Surf the Internet](https://mlsanity.com/paper/2102.12205/) (2021)
- [A metric learning approach for endoscopic kidney stone identification](https://mlsanity.com/paper/2307.07046/) (2023)

## Abstract (arXiv)

Real-world contains an overwhelmingly large number of object classes, learning all of which at once is infeasible. Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions quickly with only a few samples. Recent works [7, 41] show that simply learning a good feature embedding can outperform more sophisticated meta-learning and metric learning algorithms for few-shot learning. In this paper, we propose a simple approach to improve the representation capacity of deep neural networks for few-shot learning tasks. We follow a two-stage learning process: First, we train a neural network to maximize the entropy of the feature embedding, thus creating an optimal output manifold using a self-supervised auxiliary loss. In the second stage, we minimize the entropy on feature embedding by bringing self-supervised twins together, while constraining the manifold with student-teacher distillation. Our experiments show that, even in the first stage, self-supervision can outperform current state-of-the-art methods, with further gains achieved by our second stage distillation process. Our codes are available at: https://github.com/brjathu/SKD.