mlsanityData to 7 Oct 2026

How to Fine-Tune Vision Models with SGD

arXiv 2211.09359 ↗First posted 17 Nov 2022cs.CV cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

SGD and AdamW are the two most used optimizers for fine-tuning large neural networks in computer vision. When the two methods perform the same, SGD is preferable because it uses less memory (12 bytes/parameter with momentum and 8 bytes/parameter without) than AdamW (16 bytes/parameter). However, on a suite of downstream tasks, especially those with distribution shifts, we find that fine-tuning with AdamW performs substantially better than SGD on modern Vision Transformer and ConvNeXt models. We find that large gaps in performance between SGD and AdamW occur when the fine-tuning gradients in the first "embedding" layer are much larger than in the rest of the model. Our analysis suggests an easy fix that works consistently across datasets and models: freezing the embedding layer (less than 1% of the parameters) leads to SGD with or without momentum performing slightly better than AdamW while using less memory (e.g., on ViT-L, SGD uses 33% less GPU memory). Our insights result in state-of-the-art accuracies on five popular distribution shift benchmarks: WILDS-FMoW, WILDS-Camelyon, BREEDS-Living-17, Waterbirds, and DomainNet.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

37 of 49 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2022A ConvNet for the 2020scited by 3,545
  12. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
25 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  3. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  9. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  10. 2018Do Better ImageNet Models Transfer Better?cited by 560
  11. 2017Large Batch Training of Convolutional Networkscited by 529
  12. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  13. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  14. 2017Functional Map of the Worldcited by 338
  15. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  16. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  17. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 166
  18. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  19. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  20. 2021Model-Based Domain Generalizationcited by 103
  21. 2017Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuningcited by 48
  22. 2022Are Vision Transformers Robust to Spurious Correlations?cited by 36
  23. 2019Targeted transfer learning to improve performance in small medical physics datasetscited by 11
  24. 2019Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networkscited by 7
  25. 2019Distilling BERT into Simple Neural Networks with Unlabeled Transfer Datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 6; 2025: 2; 2026: 2.

23 papers.

Sort:

By area:

  1. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  2. 2023Differentially Private Diffusion Models Generate Useful Synthetic Imagescited by 85
  3. 2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
  4. 2023On the Implicit Bias of Adamcited by 25
  5. 2023Out-of-Domain Robustness via Targeted Augmentationscited by 24
  6. 2023Trainable Transformer in Transformercited by 17
  7. 2023PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMscited by 14
  8. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 14
  9. 2022A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimizationcited by 11
  10. 2023Contextual Vision Transformers for Robust Representation Learningcited by 10
  11. 2023A Holistic Assessment of the Reliability of Machine Learning Systemscited by 9
  12. 2024Fusion Transformer with Object Mask Guidance for Image Forgery Analysiscited by 7
11 more
  1. 2023Domain adaptation using optimal transport for invariant learning using histopathology datasetscited by 4
  2. 2023RetailKLIP : Finetuning OpenCLIP backbone using metric learning on a single GPU for Zero-shot retail product image classificationcited by 3
  3. 2024AutoFT: Learning an Objective for Robust Fine-Tuningcited by 3
  4. 2024Learning Embeddings with Centroid Triplet Loss for Object Identification in Robotic Graspingcited by 3
  5. 2024One Step Learning, One Step Reviewcited by 2
  6. 2024Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systemscited by 1
  7. 2026Beyond a Single Explanation of the Adam--SGD Gapcited by 1
  8. 2023Improving Reliability of Fine-tuning with Block-wise Optimisationcited by 0
  9. 2023Can LLMs Configure Software Toolscited by 0
  10. 2025When Do Domain-Specific Foundation Models Justify Their Cost? A Systematic Evaluation Across Retinal Imaging Taskscited by 0
  11. 2026What to Preserve, Where to Adapt: A Depth-Wise Analysis of Forgetting in Continual Gynecological Image Segmentationcited by 0