Data to 14 Oct 2026

Surgical Fine-Tuning Improves Adaptation to Distribution Shifts

arXiv 2210.11466 ↗First posted 20 Oct 2022cs.LG cs.AIcited by 205

Not on the map. Why?

Abstract (arXiv)

A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model, preserving learned features while also adapting to the new task. This paper shows that in such settings, selectively fine-tuning a subset of layers (which we term surgical fine-tuning) matches or outperforms commonly used fine-tuning approaches. Moreover, the type of distribution shift influences which subset is more effective to tune: for example, for image corruptions, fine-tuning only the first few layers works best. We validate our findings systematically across seven real-world data tasks spanning three types of distribution shifts. Theoretically, we prove that for two-layer neural networks in an idealized setting, first-layer tuning can outperform fine-tuning all layers. Intuitively, fine-tuning more parameters on a small target dataset can cause information learned during pre-training to be forgotten, and the relevant information depends on the type of shift.

Key facts. Cited by 205 papers in the map (data through 14 Oct 2026). Builds on 67 of its 99 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

67 of 99 references.

References · 67

2015Deep Residual Learning for Image Recognitioncited by 52,096
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
2016Overcoming catastrophic forgetting in neural networkscited by 4,787
2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
2019Invariant Risk Minimizationcited by 1,854
2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247

38 more

2014How transferable are features in deep neural networks?cited by 1,225
2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 600
2018Do Better ImageNet Models Transfer Better?cited by 565
2015Causal inference using invariant prediction: identification and confidence intervalscited by 536
2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 377
2020What is being transferred in transfer learning?cited by 368
2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
2016Unsupervised Domain Adaptation with Residual Transfer Networkscited by 341
2017Functional Map of the Worldcited by 338
2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
2020Environment Inference for Invariant Learningcited by 306
2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
2018What is the Effect of Importance Weighting in Deep Learning?cited by 251
2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 227
2021WARP: Word-level Adversarial ReProgrammingcited by 221
2022Test-Time Training with Masked Autoencoderscited by 212
2020Adaptive Risk Minimization: Learning to Adapt to Domain Shiftcited by 183
2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
2021A Fine-Grained Analysis on Distribution Shiftcited by 171
2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 168
2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
2020Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANscited by 132
20223D Common Corruptions and Data Augmentationcited by 119
2020Test-Time Adaptable Neural Networks for Robust Medical Image Segmentationcited by 102
2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
2019Are All Layers Created Equal?cited by 94
2022Three things everyone should know about Vision Transformerscited by 90
2019What Would Elsa Do? Freezing Layers During Transformer Fine-Tuningcited by 89

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 65; 2024: 61; 2025: 38; 2026: 40.

205 papers.

Sort:

By area:

2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 271
2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
2023Mitigating the Alignment Tax of RLHFcited by 183
2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 105
2023EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Modelscited by 98
2024Model Stock: All we need is just a few fine-tuned modelscited by 92
2023Offsite-Tuning: Transfer Learning without Full Modelcited by 78
2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
2023Simple and Fast Group Robustness by Automatic Feature Reweightingcited by 68

38 more

2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
2023PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalizationcited by 59
2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 53
2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
2024SpectralEarth: Training Hyperspectral Foundation Models at Scalecited by 42
2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 41
2023Trainable Projected Gradient Method for Robust Fine-tuningcited by 37
2024Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspectivecited by 36
2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 35
2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 34
2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
2024FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuningcited by 27
2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
2024Any-Shift Prompting for Generalization over Distributionscited by 24
2024SoMA: Singular Value Decomposed Minor Components Adaptation for Domain Generalizable Representation Learningcited by 24
2023Improving Generalization of Adversarial Training via Robust Critical Fine-Tuningcited by 23
2023PockEngine: Sparse and Efficient Fine-tuning in a Pocketcited by 23
2023Adaptive Test-Time Personalization for Federated Learningcited by 23
2024Understanding Linear Probing then Fine-tuning Language Models from NTK Perspectivecited by 22
2023ClusT3: Information Invariant Test-Time Trainingcited by 21
2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
2023A Closer Look at Model Adaptation using Feature Distortion and Simplicity Biascited by 20
2023Fast Trainable Projection for Robust Fine-Tuningcited by 19
2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
2023ERM++: An Improved Baseline for Domain Generalizationcited by 18
2023Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and Howcited by 18
2023Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networkscited by 17
2024Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgettingcited by 17
2023Less is More: Selective Layer Finetuning with SubTuningcited by 14
2023Layer-wise Auto-Weighting for Non-Stationary Test-Time Adaptationcited by 14
2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
2024A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysiscited by 14
2024Comparison of fine-tuning strategies for transfer learning in medical image classificationcited by 14

50 most cited of 205 All 205 (JSON)