mlsanityData to 7 Oct 2026

DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection

arXiv 2406.00345 ↗First posted 1 Jun 2024cs.CV cs.LGcited by 14

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language models (VLMs), such as CLIP, have demonstrated impressive zero-shot capabilities for various downstream tasks. Their performance can be further enhanced through few-shot prompt tuning methods. However, current studies evaluate the performance of learned prompts separately on base and new classes. This evaluation lacks practicality for real-world applications since downstream tasks cannot determine whether the data belongs to base or new classes in advance. In this paper, we explore a problem setting called Open-world Prompt Tuning (OPT), which involves tuning prompts on base classes and evaluating on a combination of base and new classes. By introducing Decomposed Prompt Tuning framework (DePT), we theoretically demonstrate that OPT can be solved by incorporating out-of-distribution detection into prompt tuning, thereby enhancing the base-to-new discriminability. Based on DePT, we present a novel prompt tuning approach, namely, Decomposed Context Optimization (DeCoOp), which introduces new-class detectors and sub-classifiers to further enhance the base-class and new-class discriminability. Experimental results on 11 benchmark datasets validate the effectiveness of DePT and demonstrate that DeCoOp outperforms current state-of-the-art methods, providing a significant 2% average accuracy improvement.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 57 references that are in the map. First posted on arXiv 1 Jun 2024.

Builds on

30 of 57 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning31 Mar
  2. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts14 Jul
  3. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?9 Jun
  4. 2023Black Box Few-Shot Adaptation for Vision-Language models4 Apr
  5. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  6. 2022Prompt Distribution Learning6 May
  7. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  8. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2021Robust fine-tuning of zero-shot modelscited by 785
  12. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
10 more
  1. 2021Online Continual Learning in Image Classification: An Empirical Surveycited by 242
  2. 2022Patching open-vocabulary models by interpolating weightscited by 187
  3. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  4. 2019Unsupervised Out-of-Distribution Detection by Maximum Classifier Discrepancycited by 112
  5. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  6. 2021Adaptivity and Non-stationarity: Problem-dependent Dynamic Regret for Online Convex Optimizationcited by 66
  7. 2021Mixture Outlier Exposure: Towards Out-of-Distribution Detection in Fine-grained Environmentscited by 65
  8. 2023Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurtscited by 58
  9. 2024CLIP-driven Outliers Synthesis for few-shot OOD detectioncited by 6
  10. 2022LAMDA-SSL: Semi-Supervised Learning in Pythoncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 2.

14 papers.

Sort:

By area:

  1. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  2. 2025OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuningcited by 12
  3. 2024Robust Semi-Supervised Learning in Open Environmentscited by 8
  4. 2024VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoningcited by 6
  5. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  6. 2025LIFT+: Lightweight Fine-Tuning for Long-Tail Learningcited by 5
  7. 2025BMIP: Bi-directional Modality Interaction Prompt Learning for VLMcited by 2
  8. 2025CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimizationcited by 1
  9. 2025Delta Energy: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalizationcited by 1
  10. 2026NeSy-Route: A Neuro-Symbolic Benchmark for Constrained Route Planning in Remote Sensingcited by 1
  11. 2026Move-Then-Operate: Behavioral Phasing for Human-Like Robotic Manipulationcited by 1
  12. 2025CGI: Identifying Conditional Generative Models with Example Imagescited by 0
2 more
  1. 2025Boosting Open Set Recognition Performance through Modulated Representation Learningcited by 0
  2. 2025Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LPT: Less-overfitting Prompt Tuning for Vision-Language Model
  2. 2024LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models
  3. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models
  4. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
  5. 2023Distribution-Aware Prompt Tuning for Vision-Language Models
  6. 2022Unified Vision and Language Prompt Learning
  7. 2023Towards Robust Prompts on Vision-Language Models
  8. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?