mlsanityData to 14 Oct 2026

Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models

arXiv 2411.01713 ↗First posted 3 Nov 2024cs.LG cs.CL cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Modern optimizers such as AdamW, equipped with momentum and adaptive learning rate, are designed to escape local minima and explore the vast parameter space. This exploration is beneficial for finding good loss basins when training from scratch. It is not necessarily ideal when resuming from a powerful foundation model because it can lead to large deviations from the pre-trained initialization and, consequently, worse robustness and generalization. At the same time, strong regularization on all parameters can lead to under-fitting. We hypothesize that selectively regularizing the parameter space is the key to fitting and retraining the pre-trained knowledge. This paper proposes a new weight decay technique, Selective Projection Decay (SPD), that selectively imposes a strong penalty on certain layers while allowing others to change freely. Intuitively, SPD expands and contracts the parameter search space for layers with consistent and inconsistent loss reduction, respectively. Experimentally, when equipped with SPD, Adam consistently provides better in-distribution generalization and out-of-distribution robustness performance on multiple popular vision and language benchmarks. Code available at~urlhttps://github.com/GT-RIPL/Selective-Projection-Decay.git

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 45 of its 55 references that are in the map. First posted on arXiv 3 Nov 2024.

Builds on

45 of 55 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
33 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  2. 2019Towards VQA Models That Can Readcited by 1,718
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  4. 2019Natural Adversarial Examplescited by 1,408
  5. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  7. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  8. 2016An overview of gradient descent optimization algorithmscited by 1,033
  9. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  10. 2021Robust fine-tuning of zero-shot modelscited by 797
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  12. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  13. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  14. 2017Large Batch Training of Convolutional Networkscited by 535
  15. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  16. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  17. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 283
  18. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  19. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 227
  20. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  21. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  22. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  23. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  24. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
  25. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  26. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  27. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  28. 2021Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answeringcited by 71
  29. 2022Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Taskscited by 52
  30. 2020Distance-Based Regularisation of Deep Networks for Fine-Tuningcited by 48
  31. 2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37
  32. 2023Fast Trainable Projection for Robust Fine-Tuningcited by 19
  33. 2020Guarantees for Tuning the Step Size using a Learning-to-Learn Approachcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Cautious Weight Decaycited by 13
  2. 2025Directional Gradient Projection for Robust Fine-Tuning of Foundation Modelscited by 11
  3. 2025MAPS: Preserving Vision-Language Representations via Module-Wise Proximity Scheduling for Better Vision-Language-Action Generalizationcited by 7
  4. 2025FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answeringcited by 5
  5. 2026Linearization Explains Fine-Tuning in Large Language Modelscited by 2
  6. 2026Restoring Neural Network Plasticity for Faster Transfer Learningcited by 0
  7. 2026The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Modelscited by 0
  8. 2026Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuningcited by 0
  9. 2026Fine-Tuning LLMs for Translation: General Forgetting Mitigation Does Not Preserve MT-Specific Instruction Followingcited by 0