mlsanityData to 7 Oct 2026

Fast Trainable Projection for Robust Fine-Tuning

arXiv 2310.19182 ↗First posted 29 Oct 2023cs.CVcited by 18

Not on the map. Why?

Abstract (arXiv)

Robust fine-tuning aims to achieve competitive in-distribution (ID) performance while maintaining the out-of-distribution (OOD) robustness of a pre-trained model when transferring it to a downstream task. Recently, projected gradient descent has been successfully used in robust fine-tuning by constraining the deviation from the initialization of the fine-tuned model explicitly through projection. However, algorithmically, two limitations prevent this method from being adopted more widely, scalability and efficiency. In this paper, we propose a new projection-based fine-tuning algorithm, Fast Trainable Projection (FTP) for computationally efficient learning of per-layer projection constraints, resulting in an average 35% speedup on our benchmarks compared to prior works. FTP can be combined with existing optimizers such as AdamW, and be used in a plug-and-play fashion. Finally, we show that FTP is a special instance of hyper-optimizers that tune the hyper-parameters of optimizers in a learnable manner through nested differentiation. Empirically, we show superior robustness on OOD datasets, including domain shifts and natural corruptions, across four different vision tasks with five different pre-trained models. Additionally, we demonstrate that FTP is broadly applicable and beneficial to other learning scenarios such as low-label and continual learning settings thanks to its easy adaptability. The code will be available at https://github.com/GT-RIPL/FTP.git.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 52 of its 60 references that are in the map. First posted on arXiv 29 Oct 2023.

Builds on

52 of 60 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2016Learning without Forgettingcited by 2,465
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2019Natural Adversarial Examplescited by 1,401
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  13. 2021Learning to Prompt for Continual Learningcited by 938
  14. 2021Robust fine-tuning of zero-shot modelscited by 785
  15. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  16. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  17. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  18. 2018Rethinking ImageNet Pre-trainingcited by 376
  19. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  20. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  21. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  22. 2016Hyperparameter optimization with approximate gradientcited by 231
  23. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  24. 2016FractalNet: Ultra-Deep Neural Networks without Residualscited by 216
  25. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  26. 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 166
  27. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  28. 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 117
  29. 2019Overcoming Catastrophic Forgetting with Unlabeled Data in the Wildcited by 111
  30. 2020Training robust neural networks using Lipschitz boundscited by 98
  31. 2019DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networkscited by 80
  32. 2021Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Boundscited by 64
  33. 2022A Closer Look at Rehearsal-Free Continual Learningcited by 54
  34. 2022Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Taskscited by 52
  35. 2020Distance-Based Regularisation of Deep Networks for Fine-Tuningcited by 47
  36. 2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37
  37. 2021Avoiding Inference Heuristics in Few-shot Prompt-based Finetuningcited by 23
  38. 2021Partial transfusion: on the expressive influence of trainable batch norm parameters for transfer learningcited by 15

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 9; 2026: 2.

18 papers.

Sort:

By area:

  1. 2024Model Stock: All we need is just a few fine-tuned modelscited by 87
  2. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  3. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  4. 2024PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularizationcited by 14
  5. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
  6. 2025Directional Gradient Projection for Robust Fine-Tuning of Foundation Modelscited by 10
  7. 2024Rethinking Weight Decay for Robust Fine-Tuning of Foundation Modelscited by 9
  8. 2025PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocationcited by 9
  9. 2025FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answeringcited by 5
  10. 2025MAPS: Preserving Vision-Language Representations via Module-Wise Proximity Scheduling for Better Vision-Language-Action Generalizationcited by 5
  11. 2025Mixture of Reasonings: Teach Large Language Models to Reason with Adaptive Strategiescited by 3
  12. 2024Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Modelscited by 2
6 more
  1. 2025VeFA: Vector-Based Feature Space Adaptation for Robust Model Fine-Tuningcited by 2
  2. 2025LARGO: Low-Rank Regulated Gradient Projection for Robust Parameter Efficient Fine-Tuningcited by 0
  3. 2025Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performancecited by 0
  4. 2025Revisiting Mixout: An Overlooked Path to Robust Finetuningcited by 0
  5. 2026The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Modelscited by 0
  6. 2026TRACER: Persistent Regularization for Robust Multimodal Finetuningcited by 0