Few-Shot Image Classification Benchmarks are Too Far From Reality: Build Back Better with Semantic Task Sampling
Not on the map. Why?
Abstract (arXiv)
Every day, a new method is published to tackle Few-Shot Image Classification, showing better and better performances on academic benchmarks. Nevertheless, we observe that these current benchmarks do not accurately represent the real industrial use cases that we encountered. In this work, through both qualitative and quantitative studies, we expose that the widely used benchmark tieredImageNet is strongly biased towards tasks composed of very semantically dissimilar classes e.g. bathtub, cabbage, pizza, schipperke, and cardoon. This makes tieredImageNet (and similar benchmarks) irrelevant to evaluate the ability of a model to solve real-life use cases usually involving more fine-grained classification. We mitigate this bias using semantic information about the classes of tieredImageNet and generate an improved, balanced benchmark. Going further, we also introduce a new benchmark for Few-Shot Image Classification using the Danish Fungi 2020 dataset. This benchmark proposes a wide variety of evaluation tasks with various fine-graininess. Moreover, this benchmark includes many-way tasks (e.g. composed of 100 classes), which is a challenging setting yet very common in industrial applications. Our experiments bring out the correlation between the difficulty of a task and the semantic similarity between its classes, as well as a heavy performance drop of state-of-the-art methods on many-way few-shot classification, raising questions about the scaling abilities of these methods. We hope that our work will encourage the community to further question the quality of standard evaluation processes and their relevance to real-life applications.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 10 May 2022.
Builds on
20 of 35 references.
References · 20
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2017Prototypical Networks for Few-shot Learningcited by 3,001
- 2016Matching Networks for One Shot Learningcited by 2,246
- 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
- 2019A Closer Look at Few-shot Classificationcited by 679
- 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
- 2018Meta-learning with differentiable closed-form solverscited by 395
- 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
- 2019A Baseline for Few-Shot Image Classificationcited by 298
- 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
- 2019Prototype Rectification for Few-Shot Learningcited by 96
- 2020Leveraging the Feature Distribution in Transfer-based Few-Shot Learningcited by 72
8 more
- 2020Transductive Information Maximization For Few-Shot Learningcited by 44
- 2020Revisiting Pose-Normalization for Fine-Grained Few-Shot Recognitioncited by 30
- 2020Adaptive Task Sampling for Meta-Learningcited by 29
- 2021Danish Fungi 2020 -- Not Just Another Image Recognition Datasetcited by 27
- 2020Variational Feature Disentangling for Fine-Grained Few-Shot Classificationcited by 26
- 2019Adaptive Posterior Learning: few-shot learning with a surprise-based memory modulecited by 25
- 2021Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmarkcited by 16
- 2021Similarity of Classification Taskscited by 7
Built on by
Citing papers per year: 2025: 1.
1 paper.