AutoFT: Learning an Objective for Robust Fine-Tuning
Not on the map. Why?
Abstract (arXiv)
Foundation models encode rich representations that can be adapted to downstream tasks by fine-tuning. However, fine-tuning a model on one data distribution often degrades performance under distribution shifts. Current approaches to robust fine-tuning use hand-crafted regularization techniques to constrain the fine-tuning process towards the pretrained model. Yet, it is hard to specify how to adapt relevant characteristics of the foundation model during fine-tuning, as this depends on how the pre-training, fine-tuning, and test data distributions relate to each other. We propose AutoFT, a data-driven approach for robust fine-tuning. Given a task, AutoFT searches for a fine-tuning procedure that enhances out-of-distribution (OOD) generalization. Specifically, AutoFT uses bi-level optimization to search for an objective function and hyperparameters that maximize post-adaptation performance on a small OOD validation set. We evaluate AutoFT on nine natural distribution shifts. Our experiments show that AutoFT significantly improves generalization to OOD inputs, outperforming existing robust fine-tuning methods. Notably, AutoFT achieves a new state-of-the-art on the WILDS iWildCam and FMoW benchmarks, outperforming the previous best methods by 6.0% and 1.5%, respectively.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 77 of its 99 references that are in the map. First posted on arXiv 18 Jan 2024.
Builds on
77 of 99 references.
References · 77
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
- 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2019Invariant Risk Minimizationcited by 1,854
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
- 2018DARTS: Differentiable Architecture Searchcited by 1,798
- 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
38 more
- 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
- 2019Natural Adversarial Examplescited by 1,408
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
- 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
- 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
- 2020In Search of Lost Domain Generalizationcited by 1,054
- 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
- 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
- 2021Robust fine-tuning of zero-shot modelscited by 797
- 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 763
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
- 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
- 2016Learning to learn by gradient descent by gradient descentcited by 620
- 2014Deep Domain Confusion: Maximizing for Domain Invariancecited by 600
- 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
- 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
- 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
- 2023Symbolic Discovery of Optimization Algorithmscited by 434
- 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
- 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
- 2017Functional Map of the Worldcited by 338
- 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
- 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 316
- 2020Environment Inference for Invariant Learningcited by 306
- 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
- 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
- 2018What is the Effect of Importance Weighting in Deep Learning?cited by 251
- 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
- 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 227
- 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
- 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
- 2020Adaptive Risk Minimization: Learning to Adapt to Domain Shiftcited by 183
- 2021A Fine-Grained Analysis on Distribution Shiftcited by 171
- 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 168
- 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
- 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
- 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
50 most cited of 77 All 77 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.