mlsanityData to 7 Oct 2026

Robust fine-tuning of zero-shot models

arXiv 2109.01903 ↗First posted 4 Sep 2021cs.CV cs.LGcited by 785

Not on the map. Why?

Cited by 175 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large pre-trained models such as CLIP or ALIGN offer consistent accuracy across a range of data distributions when performing zero-shot inference (i.e., without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts. We address this tension by introducing a simple and effective method for improving robustness while fine-tuning: ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT). Compared to standard fine-tuning, WiSE-FT provides large accuracy improvements under distribution shift, while preserving high accuracy on the target distribution. On ImageNet and five derived distribution shifts, WiSE-FT improves accuracy under distribution shift by 4 to 6 percentage points (pp) over prior work while increasing ImageNet accuracy by 1.6 pp. WiSE-FT achieves similarly large robustness gains (2 to 23 pp) on a diverse set of six further distribution shifts, and accuracy gains of 0.8 to 3.3 pp compared to standard fine-tuning on seven commonly used transfer learning datasets. These improvements come at no additional computational cost during fine-tuning or inference.

Key facts. Cited by 785 papers in the map (data through 7 Oct 2026). Builds on 94 of its 146 references that are in the map. First posted on arXiv 4 Sep 2021.

Builds on

94 of 146 references.

References · 94

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  10. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  13. 2013Describing Textures in the Wildcited by 2,110
  14. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  15. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  16. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  17. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  18. 2016Concrete Problems in AI Safetycited by 1,582
  19. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  20. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  21. 2019Natural Adversarial Examplescited by 1,401
  22. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  23. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  24. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  25. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  26. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  27. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  28. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  29. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  30. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  31. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  32. 2021Scaling Vision Transformerscited by 993
  33. 2019When Does Label Smoothing Help?cited by 985
  34. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  35. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  36. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  37. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  38. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 86; 2023: 193; 2024: 207; 2025: 171; 2026: 119.

785 papers.

Sort:

By area:

  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  10. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  12. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
38 more
  1. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  2. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  4. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  5. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  6. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  7. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  8. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  9. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 355
  10. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  11. 2022Visual Classification via Description from Large Language Modelscited by 331
  12. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  13. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  14. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  15. 2022Matryoshka Representation Learningcited by 268
  16. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  17. 2023Data Filtering Networkscited by 247
  18. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  19. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  20. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  21. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  22. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  23. 2023CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentationcited by 215
  24. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  25. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
  26. 2023Knowledge Editing for Large Language Models: A Surveycited by 189
  27. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  28. 2022Patching open-vocabulary models by interpolating weightscited by 187
  29. 2023Mitigating the Alignment Tax of RLHFcited by 181
  30. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  31. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 178
  32. 2021Meta-learning via Language Model In-context Tuningcited by 163
  33. 2023LaDI-VTON: Latent Diffusion Textual-Inversion Enhanced Virtual Try-Oncited by 151
  34. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  35. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  36. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  37. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  38. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145

50 most cited of 785 All 785 (JSON)