# Self-Supervision Can Be a Good Few-Shot Learner

arXiv: https://arxiv.org/abs/2207.09176 · HTML: https://mlsanity.com/paper/2207.09176/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 72 of its 90 references that are in the map.
- First posted on arXiv 19 Jul 2022.

## Builds on (most cited first, 30 of 72)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)

## Built on by (most cited first, 19 of 19)

- [Supervised Masked Knowledge Distillation for Few-Shot Transformers](https://mlsanity.com/paper/2303.15466/) (2023)
- [LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition](https://mlsanity.com/paper/2403.08161/) (2024)
- [Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation](https://mlsanity.com/paper/2301.01156/) (2023)
- [A Survey of the Self Supervised Learning Mechanisms for Vision Transformers](https://mlsanity.com/paper/2408.17059/) (2024)
- [GeNIe: Generative Hard Negative Images Through Diffusion](https://mlsanity.com/paper/2312.02548/) (2023)
- [Cross-Domain Few-Shot Learning via Adaptive Transformer Networks](https://mlsanity.com/paper/2401.13987/) (2024)
- [Instance-based Max-margin for Practical Few-shot Recognition](https://mlsanity.com/paper/2305.17368/) (2023)
- [BECLR: Batch Enhanced Contrastive Few-Shot Learning](https://mlsanity.com/paper/2402.02444/) (2024)
- [Benchmarking Spurious Bias in Few-Shot Image Classifiers](https://mlsanity.com/paper/2409.02882/) (2024)
- [Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyond](https://mlsanity.com/paper/2406.02978/) (2024)
- [Meta-DM: Applications of Diffusion Models on Few-Shot Learning](https://mlsanity.com/paper/2305.08092/) (2023)
- [Task-Oriented Channel Attention for Fine-Grained Few-Shot Classification](https://mlsanity.com/paper/2308.00093/) (2023)
- [Cross-Level Distillation and Feature Denoising for Cross-Domain Few-Shot Classification](https://mlsanity.com/paper/2311.02392/) (2023)
- [Few-Shot Class-Incremental Learning with Prior Knowledge](https://mlsanity.com/paper/2402.01201/) (2024)
- [WavRx: a Disease-Agnostic, Generalizable, and Privacy-Preserving Speech Health Diagnostic Model](https://mlsanity.com/paper/2406.18731/) (2024)
- [MICM: Rethinking Unsupervised Pretraining for Enhanced Few-shot Learning](https://mlsanity.com/paper/2408.13385/) (2024)
- [Like Humans to Few-Shot Learning through Knowledge Permeation of Vision and Text](https://mlsanity.com/paper/2405.12543/) (2024)
- [A Simple Task-aware Contrastive Local Descriptor Selection Strategy for Few-shot Learning between inter class and intra class](https://mlsanity.com/paper/2408.05953/) (2024)
- [Color as the Impetus: Transforming Few-Shot Learner](https://mlsanity.com/paper/2507.22136/) (2025)

## Abstract (arXiv)

Existing few-shot learning (FSL) methods rely on training with a large labeled dataset, which prevents them from leveraging abundant unlabeled data. From an information-theoretic perspective, we propose an effective unsupervised FSL method, learning representations with self-supervision. Following the InfoMax principle, our method learns comprehensive representations by capturing the intrinsic structure of the data. Specifically, we maximize the mutual information (MI) of instances and their representations with a low-bias MI estimator to perform self-supervised pre-training. Rather than supervised pre-training focusing on the discriminable features of the seen classes, our self-supervised model has less bias toward the seen classes, resulting in better generalization for unseen classes. We explain that supervised pre-training and self-supervised pre-training are actually maximizing different MI objectives. Extensive experiments are further conducted to analyze their FSL performance with various training settings. Surprisingly, the results show that self-supervised pre-training can outperform supervised pre-training under the appropriate conditions. Compared with state-of-the-art FSL methods, our approach achieves comparable performance on widely used FSL benchmarks without any labels of the base classes.