mlsanityData to 7 Oct 2026

Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detection

arXiv 2411.03359 ↗First posted 5 Nov 2024cs.CV cs.AI cs.LGcited by 15

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Out-of-distribution (OOD) detection is crucial for deploying reliable machine learning models in open-world applications. Recent advances in CLIP-based OOD detection have shown promising results via regularizing prompt tuning with OOD features extracted from ID data. However, the irrelevant context mined from ID data can be spurious due to the inaccurate foreground-background decomposition, thus limiting the OOD detection performance. In this work, we propose a novel framework, namely, Self-Calibrated Tuning (SCT), to mitigate this problem for effective OOD detection with only the given few-shot ID data. Specifically, SCT introduces modulating factors respectively on the two components of the original learning objective. It adaptively directs the optimization process between the two tasks during training on data with different prediction uncertainty to calibrate the influence of OOD regularization, which is compatible with many prompt tuning based OOD detection methods. Extensive experiments and analyses have been conducted to characterize and demonstrate the effectiveness of the proposed SCT. The code is publicly available.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 54 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

40 of 54 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey31 Jul
  2. 2024Negative Label Guided OOD Detection with Pretrained Vision-Language Models29 Mar
  3. 2024An Empirical Study Into What Matters for Calibrating Vision-Language Models12 Feb
  4. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Models3 Nov
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  5. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  6. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  7. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  8. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  9. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 774
  10. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  11. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  12. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
22 more
  1. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  2. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
  3. 2022VOS: Learning What You Don't Know by Virtual Outlier Synthesiscited by 300
  4. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  5. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  6. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  7. 2020Universal Domain Adaptation through Self Supervisioncited by 172
  8. 2015TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Trackingcited by 165
  9. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  10. 2023Non-Parametric Outlier Synthesiscited by 129
  11. 2022POEM: Out-of-Distribution Detection with Posterior Samplingcited by 108
  12. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  13. 2022DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotationscited by 102
  14. 2023Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancementcited by 77
  15. 2016Places: An Image Database for Deep Scene Understandingcited by 76
  16. 2021On the Impact of Spurious Correlation for Out-of-distribution Detectioncited by 68
  17. 2024Learning Transferable Negative Prompts for Out-of-Distribution Detectioncited by 43
  18. 2023ID-like Prompt Learning for Few-Shot Out-of-Distribution Detectioncited by 42
  19. 2023Diversified Outlier Exposure for Out-of-Distribution Detection via Informative Extrapolationcited by 38
  20. 2023GL-MCM: Global and Local Maximum Concept Matching for Zero-Shot Out-of-Distribution Detectioncited by 36
  21. 2023Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capabilitycited by 16
  22. 2023Likelihood-Aware Semantic Alignment for Full-Spectrum Out-of-Distribution Detectioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025Adaptive Multi-prompt Contrastive Network for Few-shot Out-of-distribution Detectioncited by 8
  2. 2025Knowledge Regularized Negative Feature Tuning of Vision-Language Models for Out-of-Distribution Detectioncited by 4
  3. 2025Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimationcited by 3
  4. 2025FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detectioncited by 3
  5. 2026T-QPM: Enabling Temporal Out-Of-Distribution Detection and Domain Generalization for Vision-Language Models in Open-Worldcited by 3
  6. 2025OpenEarthSensing: Large-Scale Fine-Grained Benchmark for Open-World Remote Sensingcited by 2
  7. 2025Hierarchical Vision-Language Learning for Medical Out-of-Distribution Detectioncited by 2
  8. 2026TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Modelscited by 2
  9. 2025E-MLNet: Enhanced Mutual Learning for Universal Domain Adaptation with Sample-Specific Weightingcited by 1
  10. 2025OpenHAIV: A Framework Towards Practical Open-World Learningcited by 0
  11. 2025Retrieval-Augmented Prompt for OOD Detectioncited by 0
  12. 2025Prompt Optimization Meets Subspace Representation Learning for Few-shot Out-of-Distribution Detectioncited by 0
3 more
  1. 2026Enhancing Few-Shot Out-of-Distribution Detection via the Refinement of Foreground and Backgroundcited by 0
  2. 2026Out-of-Distribution Object Detection in Street Scenes via Synthetic Outlier Exposure and Transfer Learningcited by 0
  3. 2026DynProto: Dynamic Prototype Evolution for Out-of-Distribution Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?
  2. 2025Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection
  3. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models
  4. 2024Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection
  5. 2024CLIP-driven Outliers Synthesis for few-shot OOD detection
  6. 2024Near OOD Detection for Vision-Language Prompt Learning with Contrastive Logit Score
  7. 2025Out-of-Distribution Detection with Positive and Negative Prompt Supervision Using Large Language Models
  8. 2025A Benchmark and Evaluation for Real-World Out-of-Distribution Detection Using Vision-Language Models