# Free Lunch for Few-shot Learning: Distribution Calibration

arXiv: https://arxiv.org/abs/2101.06395 · HTML: https://mlsanity.com/paper/2101.06395/

## Key facts

- Cited by 165 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 16 Jan 2021.

## Builds on (most cited first, 21 of 21)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Low-Shot Learning from Imaginary Data](https://mlsanity.com/paper/1801.05401/) (2018)
- [Charting the Right Manifold: Manifold Mixup for Few-shot Learning](https://mlsanity.com/paper/1907.12087/) (2019)
- [Negative Margin Matters: Understanding Margin in Few-shot Classification](https://mlsanity.com/paper/2003.12060/) (2020)
- [Deep Representation Learning on Long-tailed Data: A Learnable Embedding Augmentation Perspective](https://mlsanity.com/paper/2002.10826/) (2020)
- [Implicit Semantic Data Augmentation for Deep Networks](https://mlsanity.com/paper/1909.12220/) (2019)
- [Delta-encoder: an effective sample synthesis method for few-shot object recognition](https://mlsanity.com/paper/1806.04734/) (2018)
- [An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning](https://mlsanity.com/paper/1904.08479/) (2019)
- [Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks](https://mlsanity.com/paper/1810.11730/) (2018)
- [Learning to Propagate for Graph Meta-Learning](https://mlsanity.com/paper/1909.05024/) (2019)
- [Assume, Augment and Learn: Unsupervised Few-Shot Meta-Learning via Random Labels and Data Augmentation](https://mlsanity.com/paper/1902.09884/) (2019)
- [Prototype Propagation Networks (PPN) for Weakly-supervised Few-shot Learning on Category Graph](https://mlsanity.com/paper/1905.04042/) (2019)

## Built on by (most cited first, 30 of 165)

- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners](https://mlsanity.com/paper/2303.02151/) (2023)
- [Dataset Pruning: Reducing Training Data by Examining Generalization Influence](https://mlsanity.com/paper/2205.09329/) (2022)
- [Registration based Few-Shot Anomaly Detection](https://mlsanity.com/paper/2207.07361/) (2022)
- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)
- [PartImageNet: A Large, High-Quality Dataset of Parts](https://mlsanity.com/paper/2112.00933/) (2021)
- [FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning](https://mlsanity.com/paper/2309.14062/) (2023)
- [Sample Selection with Uncertainty of Losses for Learning with Noisy Labels](https://mlsanity.com/paper/2106.00445/) (2021)
- [Empowering Graph Representation Learning with Test-Time Graph Transformation](https://mlsanity.com/paper/2210.03561/) (2022)
- [Uncertainty Estimation by Fisher Information-based Evidential Deep Learning](https://mlsanity.com/paper/2303.02045/) (2023)
- [A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation](https://mlsanity.com/paper/2402.04087/) (2024)
- [Few-Shot Class-Incremental Learning via Training-Free Prototype Calibration](https://mlsanity.com/paper/2312.05229/) (2023)
- [Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models](https://mlsanity.com/paper/2211.02219/) (2022)
- [Generating Representative Samples for Few-Shot Classification](https://mlsanity.com/paper/2205.02918/) (2022)
- [Estimating Instance-dependent Bayes-label Transition Matrix using a Deep Neural Network](https://mlsanity.com/paper/2105.13001/) (2021)
- [Attribute Surrogates Learning and Spectral Tokens Pooling in Transformers for Few-shot Learning](https://mlsanity.com/paper/2203.09064/) (2022)
- [TransDARC: Transformer-based Driver Activity Recognition with Latent Space Feature Calibration](https://mlsanity.com/paper/2203.00927/) (2022)
- [Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignments](https://mlsanity.com/paper/2207.10785/) (2022)
- [Supervised Masked Knowledge Distillation for Few-Shot Transformers](https://mlsanity.com/paper/2303.15466/) (2023)
- [Delving into Semantic Scale Imbalance](https://mlsanity.com/paper/2212.14613/) (2022)
- [LibFewShot: A Comprehensive Library for Few-shot Learning](https://mlsanity.com/paper/2109.04898/) (2021)
- [When Noisy Labels Meet Long Tail Dilemmas: A Representation Calibration Method](https://mlsanity.com/paper/2211.10955/) (2022)
- [Channel Importance Matters in Few-Shot Image Classification](https://mlsanity.com/paper/2206.08126/) (2022)
- [Alleviating the Sample Selection Bias in Few-shot Learning by Removing Projection to the Centroid](https://mlsanity.com/paper/2210.16834/) (2022)
- [Does Few-shot Learning Suffer from Backdoor Attacks?](https://mlsanity.com/paper/2401.01377/) (2023)
- [Self-Supervision Can Be a Good Few-Shot Learner](https://mlsanity.com/paper/2207.09176/) (2022)
- [ProtAugment: Unsupervised diverse short-texts paraphrasing for intent detection meta-learning](https://mlsanity.com/paper/2105.12995/) (2021)
- [Objects in Semantic Topology](https://mlsanity.com/paper/2110.02687/) (2021)
- [Learning to Learn from APIs: Black-Box Data-Free Meta-Learning](https://mlsanity.com/paper/2305.18413/) (2023)
- [TOHAN: A One-step Approach towards Few-shot Hypothesis Adaptation](https://mlsanity.com/paper/2106.06326/) (2021)

## Abstract (arXiv)

Learning from a limited number of samples is challenging since the learned model can easily become overfitted based on the biased distribution formed by only a few training examples. In this paper, we calibrate the distribution of these few-sample classes by transferring statistics from the classes with sufficient examples, then an adequate number of examples can be sampled from the calibrated distribution to expand the inputs to the classifier. We assume every dimension in the feature representation follows a Gaussian distribution so that the mean and the variance of the distribution can borrow from that of similar classes whose statistics are better estimated with an adequate number of samples. Our method can be built on top of off-the-shelf pretrained feature extractors and classification models without extra parameters. We show that a simple logistic regression classifier trained using the features sampled from our calibrated distribution can outperform the state-of-the-art accuracy on two datasets (~5% improvement on miniImageNet compared to the next best). The visualization of these generated features demonstrates that our calibrated distribution is an accurate estimation.