mlsanityData to 14 Oct 2026

How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?

arXiv 2306.06048 ↗First posted 9 Jun 2023cs.CV cs.CY cs.LGcited by 38

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent large vision-language models such as CLIP have shown remarkable out-of-distribution (OOD) detection and generalization performance. However, their zero-shot in-distribution (ID) accuracy is often limited for downstream datasets. Recent CLIP-based fine-tuning methods such as prompt learning have demonstrated significant improvements in ID classification and OOD generalization where OOD labels are available. Nonetheless, it remains unclear whether the model is reliable to semantic shifts without OOD labels. In this paper, we aim to bridge the gap and present a comprehensive study to understand how fine-tuning impact OOD detection for few-shot downstream tasks. By framing OOD detection as multi-modal concept matching, we establish a connection between fine-tuning methods and various OOD scores. Our results suggest that a proper choice of OOD scores is essential for CLIP-based fine-tuning. In particular, the maximum concept matching (MCM) score provides a promising solution consistently. We also show that prompt learning demonstrates the state-of-the-art OOD detection performance over the zero-shot counterpart.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 43 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

30 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  2. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  3. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  4. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  9. 2019Natural Adversarial Examplescited by 1,408
  10. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  11. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  12. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
12 more
  1. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  2. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 600
  3. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 514
  4. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  5. 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
  6. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  7. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 405
  8. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  9. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  10. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  11. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 216
  12. 2022POEM: Out-of-Distribution Detection with Posterior Samplingcited by 108

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 16; 2025: 13; 2026: 3.

38 papers.

Sort:

By area:

  1. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 179
  2. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  3. 2024Process Reward Model with Q-Value Rankingscited by 80
  4. 2023Dream the Impossible: Outlier Imagination with Diffusion Modelscited by 77
  5. 2024How Does Unlabeled Data Provably Help Out-of-Distribution Detection?cited by 36
  6. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  7. 2024Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advancescited by 26
  8. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Modelscited by 21
  9. 2024AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Modelscited by 16
  10. 2024DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detectioncited by 14
  11. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  12. 2023Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimizationcited by 13
26 more
  1. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  2. 2023Compositional Zero-shot Learning via Progressive Language-based Observationscited by 12
  3. 2024SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximationcited by 10
  4. 2024On the Learnability of Out-of-distribution Detectioncited by 8
  5. 2024When and How Does In-Distribution Label Help Out-of-Distribution Detection?cited by 6
  6. 2023Likelihood-Aware Semantic Alignment for Full-Spectrum Out-of-Distribution Detectioncited by 5
  7. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 5
  8. 2025Delving into Out-of-Distribution Detection with Medical Vision-Language Modelscited by 5
  9. 2025Equipping Vision Foundation Model with Mixture of Experts for Out-of-Distribution Detectioncited by 5
  10. 2023Can Pre-trained Networks Detect Familiar Out-of-Distribution Data?cited by 4
  11. 2024What If the Input is Expanded in OOD Detection?cited by 4
  12. 2025Out-of-Distribution Detection using Synthetic Data Generationcited by 4
  13. 2025Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Modelscited by 4
  14. 2025A Geometry-Based View of Mahalanobis OOD Detectioncited by 4
  15. 2024Multimodal large language model for wheat breeding: a new exploration of smart breedingcited by 2
  16. 2025Medix: Out-of-Distribution Detection from Unlabeled Wild Data via Robust Gradient Statisticscited by 2
  17. 2024GraphMU: Repairing Robustness of Graph Neural Networks via Machine Unlearningcited by 1
  18. 2025Beyond Graph Model: Reliable VLM Fine-Tuning via Random Graph Adaptercited by 1
  19. 2025Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Rankingcited by 1
  20. 2025Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalizationcited by 1
  21. 2026FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Modelcited by 1
  22. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuningcited by 0
  23. 2025Language-Assisted Feature Transformation for Anomaly Detectioncited by 0
  24. 2025ClusterMine: Robust Label-Free Visual Out-Of-Distribution Detection via Concept Mining from Text Corporacited by 0
  25. 2026DBMF: A Dual-Branch Multimodal Framework for Out-of-Distribution Detectioncited by 0
  26. 2026TOOD: Task-Aware Out-of-Distribution Score Calibration for Continual Learnerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection
  2. 2025Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations
  3. 2025FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detection
  4. 2024CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection
  5. 2024Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization
  6. 2025Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language Models
  7. 2024Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detection
  8. 2025An Empirical Analysis of VLM-based OOD Detection: Mechanisms, Advantages, and Sensitivity