# Self-Calibrated Tuning of Vision-Language Models for Out-of-Distribution Detection

arXiv: https://arxiv.org/abs/2411.03359 · HTML: https://mlsanity.com/paper/2411.03359/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 40 of its 54 references that are in the map.
- First posted on arXiv 5 Nov 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 40)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [VOS: Learning What You Don't Know by Virtual Outlier Synthesis](https://mlsanity.com/paper/2202.01197/) (2022)
- [MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space](https://mlsanity.com/paper/2105.01879/) (2021)
- [Delving into Out-of-Distribution Detection with Vision-Language Representations](https://mlsanity.com/paper/2211.13445/) (2022)
- [Extremely Simple Activation Shaping for Out-of-Distribution Detection](https://mlsanity.com/paper/2209.09858/) (2022)
- [Universal Domain Adaptation through Self Supervision](https://mlsanity.com/paper/2002.07953/) (2020)
- [TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking](https://mlsanity.com/paper/1504.06755/) (2015)
- [CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No](https://mlsanity.com/paper/2308.12213/) (2023)
- [Non-Parametric Outlier Synthesis](https://mlsanity.com/paper/2303.02966/) (2023)
- [POEM: Out-of-Distribution Detection with Posterior Sampling](https://mlsanity.com/paper/2206.13687/) (2022)
- [LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning](https://mlsanity.com/paper/2306.01293/) (2023)
- [DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2206.09541/) (2022)
- [Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement](https://mlsanity.com/paper/2310.00227/) (2023)
- [Places: An Image Database for Deep Scene Understanding](https://mlsanity.com/paper/1610.02055/) (2016)
- [On the Impact of Spurious Correlation for Out-of-distribution Detection](https://mlsanity.com/paper/2109.05642/) (2021)

## Built on by (most cited first, 15 of 15)

- [Adaptive Multi-prompt Contrastive Network for Few-shot Out-of-distribution Detection](https://mlsanity.com/paper/2506.17633/) (2025)
- [Knowledge Regularized Negative Feature Tuning of Vision-Language Models for Out-of-Distribution Detection](https://mlsanity.com/paper/2507.19847/) (2025)
- [Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation](https://mlsanity.com/paper/2502.00662/) (2025)
- [FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detection](https://mlsanity.com/paper/2507.04511/) (2025)
- [T-QPM: Enabling Temporal Out-Of-Distribution Detection and Domain Generalization for Vision-Language Models in Open-World](https://mlsanity.com/paper/2603.18481/) (2026)
- [OpenEarthSensing: Large-Scale Fine-Grained Benchmark for Open-World Remote Sensing](https://mlsanity.com/paper/2502.20668/) (2025)
- [Hierarchical Vision-Language Learning for Medical Out-of-Distribution Detection](https://mlsanity.com/paper/2508.17667/) (2025)
- [TTL: Test-time Textual Learning for OOD Detection with Pretrained Vision-Language Models](https://mlsanity.com/paper/2604.15756/) (2026)
- [E-MLNet: Enhanced Mutual Learning for Universal Domain Adaptation with Sample-Specific Weighting](https://mlsanity.com/paper/2509.09006/) (2025)
- [OpenHAIV: A Framework Towards Practical Open-World Learning](https://mlsanity.com/paper/2508.07270/) (2025)
- [Retrieval-Augmented Prompt for OOD Detection](https://mlsanity.com/paper/2508.10556/) (2025)
- [Prompt Optimization Meets Subspace Representation Learning for Few-shot Out-of-Distribution Detection](https://mlsanity.com/paper/2509.18111/) (2025)
- [Enhancing Few-Shot Out-of-Distribution Detection via the Refinement of Foreground and Background](https://mlsanity.com/paper/2601.15065/) (2026)
- [Out-of-Distribution Object Detection in Street Scenes via Synthetic Outlier Exposure and Transfer Learning](https://mlsanity.com/paper/2603.16122/) (2026)
- [DynProto: Dynamic Prototype Evolution for Out-of-Distribution Detection](https://mlsanity.com/paper/2604.23729/) (2026)

## Abstract (arXiv)

Out-of-distribution (OOD) detection is crucial for deploying reliable machine learning models in open-world applications. Recent advances in CLIP-based OOD detection have shown promising results via regularizing prompt tuning with OOD features extracted from ID data. However, the irrelevant context mined from ID data can be spurious due to the inaccurate foreground-background decomposition, thus limiting the OOD detection performance. In this work, we propose a novel framework, namely, Self-Calibrated Tuning (SCT), to mitigate this problem for effective OOD detection with only the given few-shot ID data. Specifically, SCT introduces modulating factors respectively on the two components of the original learning objective. It adaptively directs the optimization process between the two tasks during training on data with different prediction uncertainty to calibrate the influence of OOD regularization, which is compatible with many prompt tuning based OOD detection methods. Extensive experiments and analyses have been conducted to characterize and demonstrate the effectiveness of the proposed SCT. The code is publicly available.