# Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference

arXiv: https://arxiv.org/abs/2204.07305 · HTML: https://mlsanity.com/paper/2204.07305/

## Key facts

- Cited by 126 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 72 references that are in the map.
- First posted on arXiv 15 Apr 2022.

## Builds on (most cited first, 30 of 58)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Revisiting Unreasonable Effectiveness of Data in Deep Learning Era](https://mlsanity.com/paper/1707.02968/) (2017)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [How Well Do Self-Supervised Models Transfer?](https://mlsanity.com/paper/2011.13377/) (2020)

## Built on by (most cited first, 30 of 126)

- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervision](https://mlsanity.com/paper/2308.07898/) (2023)
- [StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2302.09309/) (2023)
- [A Closer Look at Few-shot Classification Again](https://mlsanity.com/paper/2301.12246/) (2023)
- [Cross-Domain Few-Shot Object Detection via Enhanced Open-Set Object Detector](https://mlsanity.com/paper/2402.03094/) (2024)
- [Improving neural network representations using human similarity judgments](https://mlsanity.com/paper/2306.04507/) (2023)
- [Strong Baselines for Parameter Efficient Few-Shot Fine-tuning](https://mlsanity.com/paper/2304.01917/) (2023)
- [SVL-Adapter: Self-Supervised Adapter for Vision-Language Pretrained Models](https://mlsanity.com/paper/2210.03794/) (2022)
- [SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification](https://mlsanity.com/paper/2211.16191/) (2022)
- [Transductive Zero-Shot and Few-Shot CLIP](https://mlsanity.com/paper/2405.18437/) (2024)
- [MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition](https://mlsanity.com/paper/2305.07214/) (2023)
- [Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2403.00567/) (2024)
- [Context-Aware Meta-Learning](https://mlsanity.com/paper/2310.10971/) (2023)
- [Exploring Incompatible Knowledge Transfer in Few-shot Image Generation](https://mlsanity.com/paper/2304.07574/) (2023)
- [Visual and Textual Prior Guided Mask Assemble for Few-Shot Segmentation and Beyond](https://mlsanity.com/paper/2308.07539/) (2023)
- [TinyTrain: Resource-Aware Task-Adaptive Sparse Training of DNNs at the Data-Scarce Edge](https://mlsanity.com/paper/2307.09988/) (2023)
- [Reliable Few-shot Learning under Dual Noises](https://mlsanity.com/paper/2506.16330/) (2025)
- [Exploring Efficient Few-shot Adaptation for Vision Transformers](https://mlsanity.com/paper/2301.02419/) (2023)
- [De-Diffusion Makes Text a Strong Cross-Modal Interface](https://mlsanity.com/paper/2311.00618/) (2023)
- [DETA: Denoised Task Adaptation for Few-Shot Learning](https://mlsanity.com/paper/2303.06315/) (2023)
- [Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation](https://mlsanity.com/paper/2301.01156/) (2023)
- [Combining Observational Data and Language for Species Range Estimation](https://mlsanity.com/paper/2410.10931/) (2024)
- [Generalization in Neural Networks: A Broad Survey](https://mlsanity.com/paper/2209.01610/) (2022)
- [Hubs and Hyperspheres: Reducing Hubness and Improving Transductive Few-shot Learning with Hyperspherical Embeddings](https://mlsanity.com/paper/2303.09352/) (2023)
- [Meta-Learning with a Geometry-Adaptive Preconditioner](https://mlsanity.com/paper/2304.01552/) (2023)
- [On the Importance of Spatial Relations for Few-shot Action Recognition](https://mlsanity.com/paper/2308.07119/) (2023)
- [LFS-GAN: Lifelong Few-Shot Image Generation](https://mlsanity.com/paper/2308.11917/) (2023)
- [LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms](https://mlsanity.com/paper/2311.11420/) (2023)
- [Foundation Models for Bioacoustics -- a Comparative Review](https://mlsanity.com/paper/2508.01277/) (2025)
- [Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification](https://mlsanity.com/paper/2206.08138/) (2022)

## Abstract (arXiv)

Few-shot learning (FSL) is an important and topical problem in computer vision that has motivated extensive research into numerous methods spanning from sophisticated meta-learning methods to simple transfer learning baselines. We seek to push the limits of a simple-but-effective pipeline for more realistic and practical settings of few-shot image classification. To this end, we explore few-shot learning from the perspective of neural network architecture, as well as a three stage pipeline of network updates under different data supplies, where unsupervised external data is considered for pre-training, base categories are used to simulate few-shot tasks for meta-training, and the scarcely labelled data of an novel task is taken for fine-tuning. We investigate questions such as: (1) How pre-training on external data benefits FSL? (2) How state-of-the-art transformer architectures can be exploited? and (3) How fine-tuning mitigates domain shift? Ultimately, we show that a simple transformer-based pipeline yields surprisingly good performance on standard benchmarks such as Mini-ImageNet, CIFAR-FS, CDFSL and Meta-Dataset. Our code and demo are available at https://hushell.github.io/pmf.