mlsanityData to 7 Oct 2026

SAFT: Towards Out-of-Distribution Generalization in Fine-Tuning

arXiv 2407.03036 ↗First posted 3 Jul 2024cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Handling distribution shifts from training data, known as out-of-distribution (OOD) generalization, poses a significant challenge in the field of machine learning. While a pre-trained vision-language model like CLIP has demonstrated remarkable zero-shot performance, further adaptation of the model to downstream tasks leads to undesirable degradation for OOD data. In this work, we introduce Sparse Adaptation for Fine-Tuning (SAFT), a method that prevents fine-tuning from forgetting the general knowledge in the pre-trained model. SAFT only updates a small subset of important parameters whose gradient magnitude is large, while keeping the other parameters frozen. SAFT is straightforward to implement and conceptually simple. Extensive experiments show that with only 0.1% of the model parameters, SAFT can significantly improve the performance of CLIP. It consistently outperforms baseline methods across several benchmarks. On the few-shot learning benchmark of ImageNet and its variants, SAFT gives a gain of 5.15% on average over the conventional fine-tuning method in OOD settings.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 53 of its 65 references that are in the map. First posted on arXiv 3 Jul 2024.

Builds on

53 of 65 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
38 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2013Describing Textures in the Wildcited by 2,110
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  6. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  7. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  13. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  14. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  15. 2021Robust fine-tuning of zero-shot modelscited by 785
  16. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  17. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  18. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  19. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  20. 2018Do Better ImageNet Models Transfer Better?cited by 560
  21. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  22. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  23. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  24. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  25. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  26. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  27. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  28. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  29. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  30. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  31. 2020The Hardware Lotterycited by 160
  32. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  33. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  34. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 125
  35. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  36. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  37. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  38. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Promptscited by 51

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 4.

12 papers.

  1. 2025Directional Gradient Projection for Robust Fine-Tuning of Foundation Modelscited by 10
  2. 2025Fine Tuning without Catastrophic Forgetting via Selective Low Rank Adaptationcited by 6
  3. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  4. 2025Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Datacited by 3
  5. 2025Robust Learning of Diverse Code Editscited by 2
  6. 2025Sparsity Outperforms Low-Rank Projections in Few-Shot Adaptationcited by 1
  7. 2025GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameterscited by 1
  8. 2025Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filteringcited by 1
  9. 2026Breaking the Limits of Open-Weight CLIP: An Optimization Framework for Self-supervised Fine-tuning of CLIPcited by 1
  10. 2026Taxonomy-aware deep learning for hierarchical marine species classification in underwater imagerycited by 0
  11. 2026LoRSA: Toward Generalizable Parameter-Efficient Fine-Tuning for Biomedical Downstream Taskscited by 0
  12. 2026Awaken, Then Scale: Tiny Adaptation for MRI Reconstructioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
  2. 2024Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts
  3. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  4. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
  5. 2024CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection
  6. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Models
  7. 2025GLAD: Generalizable Tuning for Vision-Language Models
  8. 2024Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning