Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models
Not on the map. Why?
Abstract (arXiv)
Fine-tuning foundation models often compromises their robustness to distribution shifts. To remedy this, most robust fine-tuning methods aim to preserve the pre-trained features. However, not all pre-trained features are robust and those methods are largely indifferent to which ones to preserve. We propose dual risk minimization (DRM), which combines empirical risk minimization with worst-case risk minimization, to better preserve the core features of downstream tasks. In particular, we utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of model robustness: expected performance and worst-case performance, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of CLIP ViT-L/14@336 on ImageNet (75.9 to 77.1), WILDS-iWildCam (47.1 to 51.8), and WILDS-FMoW (50.7 to 53.1); opening up new avenues for robust fine-tuning. Our code is available at https://github.com/vaynexie/DRM .
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 70 of its 89 references that are in the map. First posted on arXiv 29 Nov 2024.
Builds on
70 of 89 references.
References · 70
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023GPT-4 Technical Reportcited by 18,396
- 2023Visual Instruction Tuningcited by 7,869
- 2013Intriguing properties of neural networkscited by 5,155
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2019Invariant Risk Minimizationcited by 1,854
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
38 more
- 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
- 2019Natural Adversarial Examplescited by 1,408
- 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
- 2020In Search of Lost Domain Generalizationcited by 1,054
- 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
- 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
- 2018Robustness May Be at Odds with Accuracycited by 880
- 2021Robust fine-tuning of zero-shot modelscited by 797
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
- 2013Domain Generalization via Invariant Feature Representationcited by 623
- 2018Born Again Neural Networkscited by 546
- 2015Causal inference using invariant prediction: identification and confidence intervalscited by 536
- 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
- 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
- 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 454
- 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
- 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
- 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
- 2022Visual Classification via Description from Large Language Modelscited by 336
- 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
- 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
- 2023Demystifying CLIP Datacited by 270
- 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
- 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
- 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
- 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 249
- 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 227
- 2020Domain Generalization using Causal Matchingcited by 225
- 2020Representation Learning via Invariant Causal Mechanismscited by 199
- 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
- 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
- 2018SpotTune: Transfer Learning through Adaptive Fine-tuningcited by 168
- 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
- 2022Causality Inspired Representation Learning for Domain Generalizationcited by 138
- 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 138
- 2022Domain Generalization by Mutual-Information Regularization with Pre-trained Modelscited by 133
- 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
- 2021OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalizationcited by 86
50 most cited of 70 All 70 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.