mlsanityData to 14 Oct 2026

Towards Calibrated Robust Fine-Tuning of Vision-Language Models

arXiv 2311.01723 ↗First posted 3 Nov 2023cs.CV cs.AIcited by 35

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Improving out-of-distribution (OOD) generalization during in-distribution (ID) adaptation is a primary goal of robust fine-tuning of zero-shot models beyond naive fine-tuning. However, despite decent OOD generalization performance from recent robust fine-tuning methods, confidence calibration for reliable model output has not been fully addressed. This work proposes a robust fine-tuning method that improves both OOD accuracy and confidence calibration simultaneously in vision language models. Firstly, we show that both OOD classification and OOD calibration errors have a shared upper bound consisting of two terms of ID data: 1) ID calibration error and 2) the smallest singular value of the ID input covariance matrix. Based on this insight, we design a novel framework that conducts fine-tuning with a constrained multimodal contrastive loss enforcing a larger smallest singular value, which is further guided by the self-distillation of a moving-averaged model to achieve calibrated prediction as well. Starting from empirical evidence supporting our theoretical statements, we provide extensive experimental results on ImageNet distribution shift benchmarks that demonstrate the effectiveness of our theorem and its practical implementation.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 93 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

68 of 93 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Enabling Calibration In The Zero-Shot Inference of Large Vision-Language Models11 Mar
  2. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  4. 2023Segment Anythingcited by 8,234
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  10. 2017On Calibration of Modern Neural Networkscited by 4,960
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  5. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  6. 2019Natural Adversarial Examplescited by 1,408
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  9. 2019When Does Label Smoothing Help?cited by 988
  10. 2021Robust fine-tuning of zero-shot modelscited by 797
  11. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  12. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  13. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  14. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  15. 2017Deep Neural Networks as Gaussian Processescited by 622
  16. 2018Born Again Neural Networkscited by 546
  17. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  18. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  19. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  20. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  21. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  22. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  23. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 210
  24. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  25. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  26. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  27. 2022ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentationcited by 178
  28. 2019Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processescited by 151
  29. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 150
  30. 2018Deep Convolutional Networks as shallow Gaussian Processescited by 138
  31. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  32. 2020Calibration of Neural Networks using Splinescited by 97
  33. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  34. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  35. 2020Delving Deep into Label Smoothingcited by 94
  36. 2020Self-Distillation as Instance-Specific Label Smoothingcited by 71
  37. 2020Post-hoc Uncertainty Calibration for Domain Drift Scenarioscited by 61
  38. 2021The Power of Contrast for Feature Learning: A Theoretical Analysiscited by 53

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 13; 2026: 10.

35 papers.

Sort:

By area:

  1. 2024Model Stock: All we need is just a few fine-tuned modelscited by 92
  2. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  3. 2024DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptationcited by 19
  4. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approachcited by 16
  5. 2024Open-Vocabulary Calibration for Fine-tuned CLIPcited by 15
  6. 2024A Survey on Safe Multi-Modal Learning Systemcited by 15
  7. 2024Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detectioncited by 15
  8. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 14
  9. 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Modelscited by 9
  10. 2025ViLU: Learning Vision-Language Uncertainties for Failure Predictioncited by 9
  11. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 7
  12. 2025Calibrated Language Models and How to Find Them with Label Smoothingcited by 7
23 more
  1. 2024MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfercited by 5
  2. 2025Visual Instruction Bottleneck Tuningcited by 5
  3. 2025Bayesian Principles Improve Prompt Learning In Vision-Language Modelscited by 4
  4. 2024BayesAdapter: enhanced uncertainty estimation in CLIP few-shot adaptationcited by 3
  5. 2025Explicit Abstention Knobs for Predictable Reliability in Video Question Answeringcited by 3
  6. 2026FragileFlow: Spectral Control of Correct-but-Fragile Predictions for Foundation Model Robustnesscited by 3
  7. 2024Improving Network Interpretability via Explanation Consistency Evaluationcited by 2
  8. 2024Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Modelscited by 2
  9. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibrationcited by 2
  10. 2026SoC: Semantic Orthogonal Calibration for Test-Time Prompt Tuningcited by 2
  11. 2026Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP modelscited by 2
  12. 2024UCDR-Adapter: Exploring Adaptation of Pre-Trained Vision-Language Models for Universal Cross-Domain Retrievalcited by 1
  13. 2024SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptationcited by 1
  14. 2025StarFT: Robust Fine-tuning of Zero-shot Models via Spuriosity Alignmentcited by 1
  15. 2026Model soups need only one ingredientcited by 1
  16. 2026Similarity-as-Evidence: Calibrating Overconfident VLMs for Interpretable and Label-Efficient Medical Active Learningcited by 1
  17. 2026SURE: Semi-dense Uncertainty-REfined Feature Matchingcited by 1
  18. 2026Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Modelscited by 1
  19. 2025Domain Generalization in-the-Wild: Disentangling Classification from Domain-Aware Representationscited by 0
  20. 2025Feed Two Birds with One Scone: Exploiting Function-Space Regularization for Both OOD Robustness and ID Fine-Tuning Performancecited by 0
  21. 2025Eigen-Value: Efficient Domain-Robust Data Valuation via Eigenvalue-Based Approachcited by 0
  22. 2026TRACER: Persistent Regularization for Robust Multimodal Finetuningcited by 0
  23. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
  2. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization
  3. 2024CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection
  4. 2023Robust Fine-Tuning of Vision-Language Models for Domain Generalization
  5. 2025Confidence-calibrated covariate shift correction for few-shot classification in Vision-Language Models
  6. 2025Difference Vector Equalization for Robust Fine-tuning of Vision-Language Models
  7. 2025Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations
  8. 2026The Geometry of Robustness: Optimizing Loss Landscape Curvature and Feature Manifold Alignment for Robust Finetuning of Vision-Language Models