mlsanityData to 7 Oct 2026

Oops, I Sampled it Again: Reinterpreting Confidence Intervals in Few-Shot Learning

arXiv 2409.02850 ↗First posted 4 Sep 2024cs.LG cs.AI stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The predominant method for computing confidence intervals (CI) in few-shot learning (FSL) is based on sampling the tasks with replacement, i.e. allowing the same samples to appear in multiple tasks. This makes the CI misleading in that it takes into account the randomness of the sampler but not the data itself. To quantify the extent of this problem, we conduct a comparative analysis between CIs computed with and without replacement. These reveal a notable underestimation by the predominant method. This observation calls for a reevaluation of how we interpret confidence intervals and the resulting conclusions in FSL comparative studies. Our research demonstrates that the use of paired tests can partially address this issue. Additionally, we explore methods to further reduce the (size of the) CI by strategically sampling tasks of a specific size. We also introduce a new optimized benchmark, which can be accessed at https://github.com/RafLaf/FSL-benchmark-again

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

23 of 37 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  6. 2016Matching Networks for One Shot Learningcited by 2,246
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  10. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  12. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
11 more
  1. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  2. 2019Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examplescited by 312
  3. 2018How to train your MAMLcited by 306
  4. 2019SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learningcited by 194
  5. 2023A Closer Look at Few-shot Classification Againcited by 46
  6. 2020Adaptive Task Sampling for Meta-Learningcited by 29
  7. 2019Meta-Transfer Learning through Hard Taskscited by 23
  8. 2022EASY: Ensemble Augmented-Shot Y-shaped Learning: State-Of-The-Art Few-Shot Classification with Simple Ingredientscited by 15
  9. 2020Expert Training: Task Hardness Aware Meta-Learning for Few-Shot Classificationcited by 6
  10. 2021Uniform Sampling over Episode Difficultycited by 6
  11. 2017Negative Results in Computer Vision: A Perspectivecited by 5

Built on by

Not cited yet.