Benchmarking Low-Shot Robustness to Natural Distribution Shifts
Not on the map. Why?
Abstract (arXiv)
Robustness to natural distribution shifts has seen remarkable progress thanks to recent pre-training strategies combined with better fine-tuning methods. However, such fine-tuning assumes access to large amounts of labelled data, and the extent to which the observations hold when the amount of training data is not as high remains unknown. We address this gap by performing the first in-depth study of robustness to various natural distribution shifts in different low-shot regimes: spanning datasets, architectures, pre-trained initializations, and state-of-the-art robustness interventions. Most importantly, we find that there is no single model of choice that is often more robust than others, and existing interventions can fail to improve robustness on some datasets even if they do so in the full-shot regime. We hope that our work will motivate the community to focus on this problem of practical importance.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 84 of its 103 references that are in the map. First posted on arXiv 21 Apr 2023.
Builds on
84 of 103 references.
References · 84
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2016Layer Normalizationcited by 5,579
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2017Prototypical Networks for Few-shot Learningcited by 3,001
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
- 2016Matching Networks for One Shot Learningcited by 2,246
- 2020Shortcut Learning in Deep Neural Networkscited by 1,928
- 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,906
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
- 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
- 2019Natural Adversarial Examplescited by 1,401
- 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,323
- 2017Adversarial Discriminative Domain Adaptationcited by 1,309
- 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,139
- 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,079
- 2020In Search of Lost Domain Generalizationcited by 1,042
- 2016An overview of gradient descent optimization algorithmscited by 1,021
- 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
- 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
- 2021Robust fine-tuning of zero-shot modelscited by 785
- 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
- 2021Domain Generalization: A Surveycited by 760
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
- 2020Self-supervised Learning: Generative or Contrastivecited by 693
- 2019A Closer Look at Few-shot Classificationcited by 679
- 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
- 2021ImageNet-21K Pretraining for the Massescited by 607
- 2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 597
- 2021Domain Generalization with MixStylecited by 563
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
- 2022Continual Test-Time Domain Adaptationcited by 520
- 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
- 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
- 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
50 most cited of 84 All 84 (JSON)
Built on by
Citing papers per year: 2023: 1.
1 paper.