# Few-Shot Learning with a Strong Teacher

arXiv: https://arxiv.org/abs/2107.00197 · HTML: https://mlsanity.com/paper/2107.00197/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 91 of its 122 references that are in the map.
- First posted on arXiv 1 Jul 2021.

## Builds on (most cited first, 30 of 91)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Generalizing from a Few Examples: A Survey on Few-Shot Learning](https://mlsanity.com/paper/1904.05046/) (2019)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [TADAM: Task dependent adaptive metric for improved few-shot learning](https://mlsanity.com/paper/1805.10123/) (2018)

## Built on by (most cited first, 16 of 16)

- [Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks](https://mlsanity.com/paper/2203.17030/) (2022)
- [TeST: Test-time Self-Training under Distribution Shift](https://mlsanity.com/paper/2209.11459/) (2022)
- [Learning to Learn from APIs: Black-Box Data-Free Meta-Learning](https://mlsanity.com/paper/2305.18413/) (2023)
- [HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2301.03330/) (2023)
- [Meta-Learning with Self-Improving Momentum Target](https://mlsanity.com/paper/2210.05185/) (2022)
- [Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning](https://mlsanity.com/paper/2408.12469/) (2024)
- [VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning](https://mlsanity.com/paper/2509.25033/) (2025)
- [Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognition](https://mlsanity.com/paper/2305.07180/) (2023)
- [Task-Distributionally Robust Data-Free Meta-Learning](https://mlsanity.com/paper/2311.14756/) (2023)
- [Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch](https://mlsanity.com/paper/2405.13078/) (2024)
- [Learning to Learn with Contrastive Meta-Objective](https://mlsanity.com/paper/2410.05975/) (2024)
- [KNN Transformer with Pyramid Prompts for Few-Shot Learning](https://mlsanity.com/paper/2410.10227/) (2024)
- [Make Explicit Calibration Implicit: Calibrate Denoiser Instead of the Noise Model](https://mlsanity.com/paper/2308.03448/) (2023)
- [DVLA-RL: Dual-Level Vision-Language Alignment with Reinforcement Learning Gating for Few-Shot Learning](https://mlsanity.com/paper/2602.00795/) (2026)
- [Siamese Transformer Networks for Few-shot Image Classification](https://mlsanity.com/paper/2408.01427/) (2024)
- [Color as the Impetus: Transforming Few-Shot Learner](https://mlsanity.com/paper/2507.22136/) (2025)

## Abstract (arXiv)

Few-shot learning (FSL) aims to generate a classifier using limited labeled examples. Many existing works take the meta-learning approach, constructing a few-shot learner that can learn from few-shot examples to generate a classifier. Typically, the few-shot learner is constructed or meta-trained by sampling multiple few-shot tasks in turn and optimizing the few-shot learner's performance in generating classifiers for those tasks. The performance is measured by how well the resulting classifiers classify the test (i.e., query) examples of those tasks. In this paper, we point out two potential weaknesses of this approach. First, the sampled query examples may not provide sufficient supervision for meta-training the few-shot learner. Second, the effectiveness of meta-learning diminishes sharply with the increasing number of shots. To resolve these issues, we propose a novel meta-training objective for the few-shot learner, which is to encourage the few-shot learner to generate classifiers that perform like strong classifiers. Concretely, we associate each sampled few-shot task with a strong classifier, which is trained with ample labeled examples. The strong classifiers can be seen as the target classifiers that we hope the few-shot learner to generate given few-shot examples, and we use the strong classifiers to supervise the few-shot learner. We present an efficient way to construct the strong classifier, making our proposed objective an easily plug-and-play term to existing meta-learning based FSL methods. We validate our approach, LastShot, in combinations with many representative meta-learning methods. On several benchmark datasets, our approach leads to a notable improvement across a variety of tasks. More importantly, with our approach, meta-learning based FSL methods can outperform non-meta-learning based methods at different numbers of shots.